Make run metadata available during property execution

[dmurvihill]

🚀 Feature Request

When checking a property, pass an extra parameter at the end of the values. It can contain information such as the seed, shrink data, run number, etc.

Motivation

I am using fast-check for an end-to-end test. Each property run I create a new database in my Postgres instance. When the run fails, I would like to leave the database in place so I can investigate the database state. I would like to add some information to the database name (and possibly within the database itself) that I can cross-reference with the test output.

Example

  it.prop(
    [fc.array(arbPerson()).chain(arbCommands)],
  )(
    'should preserve invariants throughout operation',
    async (commands, runId) => {
      const databaseName = `papertrace-validation-e2e-test-${new Date().toISOString()}-run-${runId}`;
      // create the database and run the test
    },
  );

Meanwhile, I can count the run number with a variable in a larger scope. That wouldn't help me get the seed though.

[gruhn]

I think you can also set the seed from "the outside". As far as I can tell it's not randomized for every run, only for every "property execution". I.e. if you have 100 test runs for a property execution, they all have the same seed. The seed is generated here. So you could do something like:

const seed = Date.now() ^ (Math.random() * 0x100000000);

fc.assert(
  fc.asyncProperty(
     // ...
     async (....) => {
       const databaseName = `papertrace-validation-e2e-test-${seed}-run-${runId}`;
     }
  ), 
  { seed },
);

[phaux]

@gruhn But where do you get runId from?

[gruhn]

Sorry, didn't include that in the snippet. I guess you can pass it from the outer scope like @dmurvihill sugggested. Maybe even create a reusable function:

async function runTest(runId, seed) {
  await fc.assert(
    fc.asyncProperty(
       // ...
       async (...) => {
         const databaseName = `papertrace-validation-e2e-test-${seed}-run-${runId}`;
       }
    ), 
    { seed },
  );
}

[phaux]

I just did

let runId = 0

and then

const dbName = `test-${runId++}`

[dubzzz]

Sounds like a good idea. Having part of the metadata of a run potentially accessible to the user could be great for some very advanced usage.

Given the fact that they could easily open the door to run that are not really reproducible, I think we should head towards something not as simple as passing it as a parameter. I was thinking about something like fc.getRunContext. Anyway, it's definitely something that worth discussing as it could prove useful.

Generally speaking as of today I'd rather go for something such as #6197 (comment).