Introduce (mini) unit test framework

[furszy]

Early Note:
Don’t be scared by the PR’s line changes count — most of it’s just doc or part of the test framework API.

Context:
Currently, all tests run single-threaded sequentially and the library lacks the ability to specify which test (or group of tests) you would like to run. This is not only inconvenient as more tests are added but also time consuming during development and affects downstream projects that may want to parallelize the workload (such as Bitcoin-Core CI).

PR Goal:
Introduce a lightweight, extensible C89 unit test framework with no dynamic memory allocations, providing a structured way to register, execute, and report tests. The framework supports named command-line arguments in -key=value form, parallel test execution across multiple worker processes, granular test selection (selecting tests either by name or by module name), and time accumulation reports.

The introduced framework supports:

• -help or -h: display list of available commands along with their descriptions.
• -jobs=<num>: distribute tests across multiple worker processes (default: sequential if 0).
• -target=<name> or -t=<name>: run only specific tests by name; can be repeated to select multiple tests.
• -target=<module name>, -t=<module> Run all tests within a specific module (can be provided multiple times)
• -seed=<hex>: set a specific RNG seed (defaults to random if unspecified).
• -iter=<n>: specify the number of iterations.
• -print_tests: display list of available tests and modules you can run.
• -log=<0|1>: enable or disable test execution logging (default: 0 = disabled).

Beyond these features, the idea is to also make future developments smoother, as adding new tests require only a single entry in the central test registry, and new command-line options can be introduced easily by extending the framework’s parse_arg() function.

Compatibility Note:
The framework continues accepting the two positional arguments previously supported (iterations and seed), ensuring existing workflows remain intact.

Testing Notes:
Have fun. You can quickly try it through ./tests -j=<workers_num> for parallel execution or ./tests -t=<test_name> to run a specific test (call ./tests -print_tests to display all available tests and modules).

Extra Note:
I haven't checked the exhaustive tests file so far, but I will soon. For now, this only runs all tests declared in the tests binary.

Testing Results: (Current master branch vs PR in seconds)

• Raspberry Pi 5: master ~100 s → PR ~38 s (5 jobs)
• MacBook Pro M1: master ~30 s → PR ~10 s (6 jobs)

[real-or-random]

Nice!

I looked at existing unit test frameworks in the past, but nothing seemed appropriate for us. There are not that many for C, and they were either overkill or too simple (just handful of ifdefs) so they didn't add any functionality. I thought writing our own is too annoying (or I was just lazy). But the framework is ~300 lines, that seems fine to me.

[real-or-random]

see also #1211

[jonasnick]

Thanks @furszy. I played around with this a little bit. It reduces the execution time on my machine from 26 seconds to 10 seconds (-jobs=16). Very nice! Some observations:

• It would be helpful if a helptext would be output when tests is run with -h or --help.
• I think showing all the tests that have passed is a bit overkill. I'm already assuming that the tests pass if they do not show up in the output. I only need to see tests that don't pass.
• Maybe a future PR can autodetect the number of cores and set -jobs automatically by default?
• There's an -iter command line flag, but the test output shows "test count". It would be better if we were consistent.

[jonasnick]

• The available "targets" seem to be a bit arbitary. Maybe we can try to put them into groups (perhaps similar to the grouping in #1211)?

  ./tests -print_tests
  Available tests (58):
      --------------------------------------------------
      [  1] selftest_tests
      [  2] all_proper_context_tests
      [  3] all_static_context_tests
      [  4] deprecated_context_flags_test
      [  5] scratch_tests
      [  6] int128_tests
      [  7] ctz_tests
      [  8] modinv_tests
      [  9] inverse_tests
      [ 10] hsort_tests
      [ 11] sha256_known_output_tests
      [ 12] sha256_counter_tests
      [ 13] hmac_sha256_tests
      [ 14] rfc6979_hmac_sha256_tests
      [ 15] tagged_sha256_tests
      [ 16] scalar_tests
      [ 17] field_half
      [ 18] field_misc
      [ 19] field_convert
      [ 20] field_be32_overflow
      [ 21] fe_mul
      [ 22] sqr
      [ 23] sqrt
      [ 24] ge
      [ 25] gej
      [ 26] group_decompress
      [ 27] ecmult_pre_g
      [ 28] wnaf
      [ 29] point_times_order
      [ 30] ecmult_near_split_bound
      [ 31] ecmult_chain
      [ 32] ecmult_constants
      [ 33] ecmult_gen_blind
      [ 34] ecmult_const_tests
      [ 35] ecmult_multi_tests
      [ 36] ec_combine
      [ 37] endomorphism_tests
      [ 38] ec_pubkey_parse_test
      [ 39] eckey_edge_case_test
      [ 40] eckey_negate_test
      [ 41] ecdh_tests
      [ 42] ec_illegal_argument_tests
      [ 43] pubkey_comparison
      [ 44] pubkey_sort
      [ 45] random_pubkeys
      [ 46] ecdsa_der_parse
      [ 47] ecdsa_sign_verify
      [ 48] ecdsa_end_to_end
      [ 49] ecdsa_edge_cases
      [ 50] ecdsa_wycheproof
      [ 51] extrakeys_tests
      [ 52] schnorrsig_tests
      [ 53] musig_tests
      [ 54] ellswift_tests
      [ 55] secp256k1_memczero_test
      [ 56] secp256k1_is_zero_array_test
      [ 57] secp256k1_byteorder_tests
      [ 58] cmov_tests
  

[furszy]

Thanks for the review jonasnick!

Thanks @furszy. I played around with this a little bit. It reduces the execution time on my machine from 26 seconds to 10 seconds (-jobs=16). Very nice!

Awesome :). I think we can actually do even better, will do some changes.

• It would be helpful if a helptext would be output when tests is run with -h or --help.

The help message was actually already there, but for -help only (with a single - and I forgot to add it to the PR description).
I just pushed support for -h as well.

• I think showing all the tests that have passed is a bit overkill. I'm already assuming that the tests pass if they do not show up in the output. I only need to see tests that don't pass.

Sure. Will hide the logging behind a -log option (or -silent if we want the opposite behavior).
From my experience, logging sometimes helps spot regressions or areas that can be improved (like when a test suddenly takes longer than usual), and it’s also reassuring to see them run and pass when you have a large number of them.
I’ve also been thinking about adding per-test execution time loggings, might be a good opportunity to include it too.

• Maybe a future PR can autodetect the number of cores and set -jobs automatically by default?

I'm not sure we want that. Sequential execution is usually "standard" on any system because we don’t know what else the user might be running. Picking a number of parallel tasks automatically (even if it is a low number) could hang the CPU or even make it run slower than sequential if the system is overloaded.

• There's an -iter command line flag, but the test output shows "test count". It would be better if we were consistent.

Sure 👍🏼. That was carried over from the previous code; will improve it.

[furszy]

The available "targets" seem to be a bit arbitary. Maybe we can try to put them into groups (perhaps similar to the grouping in #1211)?

Yeah. Just reworked the framework to support registering and running groups of tests in a generic manner. This means we can now run specific tests and/or specific groups of tests via the -target/-t arg.

On top of that, made the framework reusable across binaries and improved the overall API (we can now easily connect the tests_exhaustive binary to it too — probably something for a follow-up), along with improvements to the consumers’ structure, enforcing consistency and a specific pattern that all consumers will follow.

Other than that, the -print_tests option was improved to display the available modules and tests.

A simple usage example:
./tests -print_tests → pick any module (like field) and run: ./tests -t=field
You can also combine this with -j=<num_workers>, plus specify multiple targets as needed.

[theStack]

Concept ACK

Tested this quickly on an arm64 machine and observed a nice >2x speedup (~24.4s j=1 vs. ~11.1s j=6), also played around running only tests of certain modules, which worked as expected. Left just a few first-look nits.

[furszy]

Updated per feedback, thanks theStack and hebasto!

[hebasto]

318460c

nit: It would be good to check return values of the pipe() and fork() calls consistently, for example ...==-1.

[furszy]

I'm not sure these two functions are directly comparable. They have different return value ranges: pipe() just indicates success or failure (0 or -1), while fork() has three cases: -1 for an error, 0 in the child process, and any value greater than 0 is the child’s PID in the parent. So it seemed slightly more accurate to me to describe the entire range for fork().

[hebasto]

6162b76:

It's unfortunate to have such a pair of confusing macros. Perhaps their naming could be improved, although I don’t have a concrete suggestion.

[furszy]

6162b76:

It's unfortunate to have such a pair of confusing macros. Perhaps their naming could be improved, although I don’t have a concrete suggestion.

A small scripted-diff would improve the situation and let us use only one of them. We just need to rename the test functions that start with "run_*" so that is not included in the name. I just tried to avoid expanding the scope of the PR further.

[hebasto]

Approach ACK 5a50106.

I've completed my first round of reviewing. The code looks good.

[furszy]

Updated per feedback, thanks Hebasto!
Moved the headers existence check to the build systems (both).

[john-moffett]

I like this a lot. Approach ACK 726e70b

[furszy]

Updated per feedback. Thanks john-moffett!
Adapted the autoconf code for better portability and fixed a misplaced targets size check.

[stratospher]

nice! useful to be able to run a subset of tests.

[stratospher]

0423532: also don't see the Iterations count = log in the CI. I guess this is related to the environment variable not being set.

maybe useful to always print the log even if it's the default value being used/-iters isn't used ?

[furszy]

0423532: also don't see the Iterations count = log in the CI. I guess this is related to the environment variable not being set.

Yeah. We only print when the arg is provided by the user.

maybe useful to always print the log even if it's the default value being used/-iters isn't used ?

We should probably unify all args prints within a single place too.

[john-moffett]

Two more nits and one observation. Otherwise all looks good.

[john-moffett]

ACK 7b51e53