Batch assertion mode support

[TomerAberbach]

🚀 Feature Request

Motivation

I'm working on a library and am trying to apply property-based testing to it. I have an assertion that is expensive to run even on a single generated value. As a result, the property-based testing is way too slow.

I was thinking about how to solve this and realized my assertion has a constant factor that makes it slow. If I could receive multiple values at the same time and assert on them together, then this would barely be slower than asserting on a single value (up to a point), and after the assertion I can deduce which values failed the assertion.

So this gave me the following idea: What if fast-check allowed you to specify "please provide me values in batches of size 50" and then the end-user is responsible for asserting on all values together and return to fast-check the values that failed.

Example

This is a sketch and doesn't have to be the final public API:

test.prop([fc.double()], { batch: 50 })(`something`, (values) => {
  const badValues = someExpensiveMethod(values)

  reportFailingValues(badValues)
  expect(badValues).toStrictEqual([])
})

---

NOTE: For now I'm doing fc.array(myArbitrary, { minLength: 1 }) and limiting numRuns to a smaller number to get a similar behavior. It's unclear to me if the behavior I'm getting from this is actually equivalent to what I'm suggesting above. I suspect with the above feature fast-check could maybe be smarter, but I'm not sure.

Curious what you think!

[gruhn]

If the assertion fails, then shrinking would still be very slow, right?

Is it true that the expense constant step does not depend on the input at all? If so, is it possible to do that step outside the test? E.g. if your method looks like this:

function someExpensiveMethod(inputs) {
   const precomputedThing = expensiveStep()
   return inputs.map(x => 
       cheapStep(precomputedThing, x)
   )
}

then the test could look like this:

const precomputedThing = expensiveStep()

test.prop(fc.double())(`something`, (value) => {
   const result = cheapStep(precomputedThing, value)
   // ...
})

That would have the benefit of also making shrinking fast.

[TomerAberbach]

Unfortunately, I can't separate out the expensive part. It is coupled to the input.

Concretely, my library is doing codegen based on some input and I'm running the TypeScript typechecker in memory on the resulting code, which is quite slow. However, I can concatenate multiple codegen'd runs together and typecheck them together (I can guarantee they won't conflict) and then look at the error diagnostics to know which runs were bad.

I'm imagining other problems could have the same shape, where you're calling out to some external thing that's slow, but you can batch inputs to it (e.g. REST APIs, CLIs, etc.)

For shrinking I was envisioning that fast-check would provide batches containing multiple longish shrinking paths (almost like branch prediction in CPUs) so that it could jump ahead in the shrinking and do it in fewer calls to the property callback

[gruhn]

Concretely, my library is doing codegen based on some input and I'm running the TypeScript typechecker in memory on the resulting code,

I see, interesting stuff.

For shrinking I was envisioning that fast-check would provide batches containing multiple longish shrinking paths (almost like branch prediction in CPUs) so that it could jump ahead in the shrinking and do it in fewer calls to the property callback

I've heard of branch predication but I'm not familiar how it's implemented. With the fc.array-based approach that you are already using, I think fast-check will try to shrink the array length first before shrinking items. Are you basically suggesting to keep the length (aka batch size) fixed and shrink the items in parallel? I think one problem with that is that fast-check only gets one bit of information for each test run. Test failed: yes or no. So it can't know which item in the batch caused the failure. fast-check still has to shrink the items individually. E.g. like here, where I fixed the array length to 20 and the failing condition is whether the array contains a 7:

√ [[2147483640,564130308,2086330806,29,3,1597145858,1426124902,24,18,486933324,1557923791,1607142269,17,2147483619,432579504,1660481337,2147483646,2147483620,17,2020306435]]
√ [[28,3,44894765,269809489,471085546,10,2147483631,706310041,12,1883120394,2147483640,579582912,731908897,29,944521005,293382282,1772029492,587131727,1295761035,17]]
√ [[2147483622,1137456576,15,19,28,976359733,1027566046,179599048,1447412991,1353775534,535645637,839901533,12,651721850,351291580,2147483640,30,2147483639,1763009712,17]]
× [[57112254,2147483628,6,1685671267,23,1182006251,1759341326,1456744187,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. × [[0,2147483628,6,1685671267,23,1182006251,1759341326,1456744187,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . × [[0,0,6,1685671267,23,1182006251,1759341326,1456744187,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . × [[0,0,0,1685671267,23,1182006251,1759341326,1456744187,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . × [[0,0,0,0,23,1182006251,1759341326,1456744187,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . × [[0,0,0,0,0,1182006251,1759341326,1456744187,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . × [[0,0,0,0,0,0,1759341326,1456744187,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . × [[0,0,0,0,0,0,0,1456744187,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . × [[0,0,0,0,0,0,0,0,283976548,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . . × [[0,0,0,0,0,0,0,0,0,2147483629,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,1222220312,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . . . . . √ [[0,0,0,0,0,0,0,0,0,0,0,0,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . . . . . √ [[0,0,0,0,0,0,0,0,0,0,0,4,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . . . . . √ [[0,0,0,0,0,0,0,0,0,0,0,6,1792177509,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,0,1676023052,633387594,19,12,1552959256,27,24]]
. . . . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,0,0,633387594,19,12,1552959256,27,24]]
. . . . . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,19,12,1552959256,27,24]]
. . . . . . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,12,1552959256,27,24]]
. . . . . . . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,1552959256,27,24]]
. . . . . . . . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,27,24]]
. . . . . . . . . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,24]]
. . . . . . . . . . . . . . . . . . . × [[0,0,0,0,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0]]

I guess to do this properly in parallel we would need special assertions that tell fast-check for which items the test failed.

[TomerAberbach]

Right, the end-user would need a way to say which items failed (that was in my suggestion above; sorry if it was not clear)

I don't know if this is true, but I assume fast-check effectively creates a path through a tree of values when it shrinks. I was imagining that fast-check would provide multiple branches (of length >1) worth of values to the property (i.e. shrinking several times in advance so it could provide multiple values), and then based on which values failed (which the property should return somehow) fast-check could jump to its next point in the shrinking (and skip over multiple values at once)

This is pretty hand-wavy because I don't actually know how fast-check works on a deep level, so I can only provide ideas based on my intuition. Feel free to let me know if this is completely infeasible or nonsensical lol

[gruhn]

Right, the end-user would need a way to say which items failed (that was in my suggestion above; sorry if it was not clear)

Ah sorry, that's what you meant with reportFailingValues(badValues). I think you would have to make that part of the assertion though. Otherwise it permits all kinds of nonsensical situations:

• call reportFailingValues multiple times
• don't call reportFailingValues but assertion still fails
• call reportFailingValues but assertion passes
• call reportFailingValues but assertion fails for "other reasons"

Also reportFailingValues has to communicate to fast-check via side effect (in contrast to assertions which throw exceptions). If multiple tests run in parallel, how does fast check know which test reported the values?

So I think we would need a bunch of custom assertions for this.

[TomerAberbach]

Yeah, I think for the vitest, jest, etc. integrations it'll be a bit awkward to get right. For plain fast-check I guess you'd just have to return an array of bad values from the property? So length zero implies "no failures"?

[gruhn]

For plain fast-check I guess you'd just have to return an array of bad values from the property? So length zero implies "no failures"?

True, that's also a simple option. Although, fast-check would probably need some extra information to associate the output with the input. If you just return an array with one element, fast-check wouldn't know which item in the input batch it corresponds to.

[TomerAberbach]

Yup yup, makes sense

Taking a step back, how do you feel about the feature idea in general? There are API and other problems to solve here, but that's all moot if you think the idea is bad or not worth the effort

For what it's worth, I am not really blocked or anything. The tests just run more slowly than ideal and I have to do the fc.array(...) + lower numRuns, but it's not too bad

[TomerAberbach]

And if you're curious, I just published the first (beta) version of the package where this is applicable

https://github.com/TomerAberbach/type-to-fast-check

I'm guessing you'll find this cool / interesting (cc @dubzzz as well)

[gruhn]

Taking a step back, how do you feel about the feature idea in general?

In my projects I usually prefer to keep extra parameters, config flags, etc. to minimum. Especially, when the problem can already be solved differently. But I don't want to make any "product decisions" here 😛 Let's see what @dubzzz has to say.

And if you're curious, I just published the first (beta) version of the package where this is applicable
https://github.com/TomerAberbach/type-to-fast-check

That looks very interesting indeed!

[dubzzz]

While I'm pretty sure some users would be interested in batching in some of their tests, my feeling is that we will make the API more complex for a limited usage.

The idea of returning the values that failed to run will not be simple to capture from a typing point of view. Users could easily provide us with either altered instances, cloned instances or crap. Then the instance alone is not enough to trigger the shrink given the shrink needs to have access to the context of the generation, meaning that we might need the index of the item rather than its value.

TLDR. Not convinced by the need (for now)