Automate code snippet tests

[ranguard]

Run tests against the code snippets.

Moose::Manual apparently has some useful examples of doing this

[doherty]

Test::Inline lets you put test snippets in the files themselves, then extract them into test files - but I think it will only extract from .pm files...

[ranguard]

I think the main problem is most of the examples will be partial code, not fully complete (as otherwise it would bloat the faq too much), so there would need to be some other way of wrapping the code snippet with whatever was needed to make it into a test. So this may be a non-starter.

[doherty]

Right - the code examples aren't themselves tests. What Test::Inline would let you do is:

=pod

Here's how you get rid of newlines when reading a file:

    while(<>) { # puts each line in $_
        chomp; # $_ is implicit
        # Now $_ lacks the newline at the end
    }

=begin testing

open my $in, '<', __FILE__ or die "Couldn't open myself: $!";
while (<$in>) {
    chomp;
    unlike $_, qr/\n$/;
}

done_testing($.);

=done testing

=cut

So, you get to write your tests beside the FAQ they relate to. You still have to write stuff as a test. That block between begin testing and end testing gets extracted, put in a test script template so it has use strict, use Test::More etc, and then dumped in t/ (which could easily be automated with Dist::Zilla).

I'm not sure how much better that is than putting the tests in t/ directly from the start. I guess with it in the POD, you're more likely to be reminded to update the test if you change the FAQs' answer?

[ranguard]

Ahh, this sounds good, we should try that going forward, Inline is much better than t/ as so many people may edit the file over time and it is easier to track.

Would be good to check DZIL doesn't add any modules used in the tests as dependencies. We'd only want these tests to run before release, rather than during normal test/install.