Add release checklist test: "Rackunit tests"

[jackfirth]

Owned by Ryan Culpepper. Command: racket -l tests/rackunit/run-tests. Some tests are expected to fail which is a blocker for automation in the Docker script.

This is different from how rackunit's tests are run in Travis CI and DrDr. Perhaps this could be replaced with raco test -p --drdr racket-lib rackunit-test instead? That takes care of the expected failures problem.

[samth]

@rmculpepper what would we need to do to be able to run the release tests automatically? Can we fix rackunit not to expect any to fail?

[rmculpepper]

I think it would make sense to just remove the running of the failure-tests from run-tests.rkt. Maybe put it in a separate file run by drdr but not travis?

[samth]

How does drdr not count that as a failure?

[samth]

For reference, here's a recent DrDr run: http://drdr.racket-lang.org/43651/racket/share/pkgs/rackunit-test/tests/rackunit/run-tests.rkt

[jackfirth]

How does drdr not count that as a failure?

Because the run-tests function does something odd I only just now discovered. If there are no failures, run-tests prints "X success(es) 0 failure(s) 0 error(s) X test(s) run" to stdout but if there are failures, each failure's message and the final count summary message is printed to stderr. Weird.

Anyway, that means the code in tests/rackunit/run-tests for switching the error port with the output port creates a test module that:

1. Prints everything to stdout when any of the expected failures fail
2. Prints nothing to stdout and prints a count message to stderr when the expected failures all pass

So if the expected failures all start passing then DrDr will notice the stderr output and fail. I don't know how this works if only one of the expected failure tests starts passing all of a sudden, but the other ones keep failing. I think DrDr will notice a change in what's printed to stdout but won't consider it a failure.

I really ought to finish racket/rackunit#59 already and use that for racket/rackunit#79, that would make all of this much simpler.

I think it would make sense to just remove the running of the failure-tests from run-tests.rkt. Maybe put it in a separate file run by drdr but not travis?

Mentioned this a bit in #27 but it's painful when some tests are run in some setups and not others in ways that aren't documented or discoverable. I'd much rather just fix those tests to not do sneaky things with stdout, stderr, or the test log in the first place.

[samth]

So does that mean that checking the exit code is sufficient here, and so Docker is all good?

[jackfirth]

I think so, yes.