feat: add schema validation matchers

[zirkelc]

Description

This PR implements a simple proof of concept for the feature suggested in #8485 to use a standard schema like Zod, Valibot and ArkType for validating data.

Three options have been implemented for further discussion:

• custom equality tester for expect.toEqual()
• custom matcher expect.toEqualSchema()
• asymmetric matcher expect.schemaMatching()

Usage:

const schema = z.object({
  name: z.string(),
  email: z.email(),
});
const validData = { name: 'John', email: '[email protected]' };

// custom equality test using toEqual
expect(validData).toEqual(schema);
expect({ email: "[email protected]" }).toEqual({ email: z.email() });

// custom matcher using toEqualSchema
expect(validData).toEqualSchema(schema);
expect({ email: "[email protected]" }).toEqualSchema({ email: z.email() });

// asymmetric matcher
expect(validData).toEqual(expect.schemaMatching(schema));
expect({ email: "[email protected]" }).toEqual(expect.schemaMatching({ email: z.email() }));

expect.toEqual(schema)

The custom equality tester for expect.toEqual() has the problem that a failing assertions prints the stringified schema as Expected value instead of the issues from the validation:

I assume there is no easy way to overwrite the message, please correct me if I'm wrong.

expect.toEqualSchema(schema)

The custom matcher for expect.toEqualSchema() has the advantage that we have full control over the assertion message in case of a failure:

However, if expect.toEqualSchema() is used inside an asymmetric matcher like expect.objectContaining() and fails, it also prints the stringified schema as Expected value like expect.toEqual():

expect.schemaMatching(schema)

The asymmetric matcher expect.schemaMatching(schema) seems to be the best candidate since it gives us full control over the assertion messages, especially when used inside another asymmetric matcher like expect.objectContaining:

---

I would lean towards implementing it as a asymmetric matcher since it allows the most customisation options and is well encapsulated from the rest and therefore reduces the risk of unknown side-effects on existing matchers likes toEqual().

Please let me know if and how you'd like to proceed.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• Ideally, include a test that fails without this PR but passes with it.
• Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.
• Please check Allow edits by maintainers to make review process faster. Note that this option is not available for repositories that are owned by Github organizations.

Tests

• Run the tests with pnpm test:ci.

Documentation

• If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[sheremet-va]

The custom equality tester for expect.toEqual() has the problem that a failing assertions prints the stringified schema as Expected value instead of the issues from the validation:

The printing is done by @vitest/pretty-format package which is a fork of pretty-format. We can just add a plugin similar to the asymmetric matchers one:

vitest/packages/pretty-format/src/plugins/AsymmetricMatcher.ts

Line 17 in de74785

export const serialize: NewPlugin['serialize'] = (

For the issues themselves, I think it's time to add something like getLastErrors that returns the message() from the asymmetric matcher:

const isEqual = equals(a, b) // keep boolean
const errors = getLastErrors() // returns errors generated during equals

Both are sync so there should be no race condition. Another thing we can do is add a new function. Something like equalsWthErrors(a, b) that returns { equals: boolean, errors: string[] } (names are just examples).

[sheremet-va]

One nice thing with wrapping it as a matcher is that we can keep the issues as the state and return the text from toString (as example):

class Matcher {
  asymmetricMatch() {
    const valid = isEqual(a, b)
    this.issues = valid.issues
    return valid.valid
  }

  toString() {
    return `Schema<${this.issues}>`
  }
}

[zirkelc]

The custom equality tester for expect.toEqual() has the problem that a failing assertions prints the stringified schema as Expected value instead of the issues from the validation:

The printing is done by @vitest/pretty-format package which is a fork of pretty-format. We can just add a plugin similar to the asymmetric matchers one:

vitest/packages/pretty-format/src/plugins/AsymmetricMatcher.ts

Line 17 in de74785

export const serialize: NewPlugin['serialize'] = (

I had the same idea, but wasn't sure if you'll want to add a schema-specific plugin to pretty-format. That's the main reason I'm preferring an asymmetric matcher like expect.schemaMatching() (or another name) because the AsymmetricMatcher already handles it.

For the issues themselves, I think it's time to add something like getLastErrors that returns the message() from the asymmetric matcher:

const isEqual = equals(a, b) // keep boolean
const errors = getLastErrors() // returns errors generated during equals

Both are sync so there should be no race condition. Another thing we can do is add a new function. Something like equalsWthErrors(a, b) that returns { equals: boolean, errors: string[] } (names are just examples).

This would definitely make sense and be a good improvement for other matchers that need more information than just pass=true/false. However, I have no intuitive feeling how much effort this would be.

One nice thing with wrapping it as a matcher is that we can keep the issues as the state and return the text from toString (as example):

class Matcher {
  asymmetricMatch() {
    const valid = isEqual(a, b)
    this.issues = valid.issues
    return valid.valid
  }

  toString() {
    return `Schema<${this.issues}>`
  }
}

I think that's what I already do with expect.schemaMatching(), or do you mean something specific? Except I'm using the toAsymmetricMatcher function to format the issues instead of toString. Would have to check why I went with toAsymmetricMatcher.

[sheremet-va]

I think that's what I already do with expect.schemaMatching(), or do you mean something specific? Except I'm using the toAsymmetricMatcher function to format the issues instead of toString. Would have to check why I went with toAsymmetricMatcher.

You are right, it is what you are already doing, I misremembered your implementation when I left a comment.

This would definitely make sense and be a good improvement for other matchers that need more information than just pass=true/false. However, I have no intuitive feeling how much effort this would be.

You are right here, we do not need it for this implementation.

Overall, I agree that expect.schemaMatching(schema) seems to be the best API out of all of them.

[zirkelc]

I think that's what I already do with expect.schemaMatching(), or do you mean something specific? Except I'm using the toAsymmetricMatcher function to format the issues instead of toString. Would have to check why I went with toAsymmetricMatcher.

You are right, it is what you are already doing, I misremembered your implementation when I left a comment.

I added the asymmetric matcher in a later commit, maybe when you checked the PR it was not committed yet.

Overall, I agree that expect.schemaMatching(schema) seems to be the best API out of all of them.

What so you think naming-wise? I tried to align it with the existing asymmetric matchers, but something shorter like it has been discussed on the issue also has its appeal. Off the top of my head, I can think of expect.satisfies(), expect.matching()/matches(), and expect.schema().

[zirkelc]

@sheremet-va I just wanted to check how we should proceed with this PR. Should I remove all other matcher except the asymmetric one, or do you want to discuss the current options with the team internally first?

[sheremet-va]

@sheremet-va I just wanted to check how we should proceed with this PR. Should I remove all other matcher except the asymmetric one, or do you want to discuss the current options with the team internally first?

Yes, please only keep the asymmetric one.

[zirkelc]

@sheremet-va I removed the matchers.

What about the naming: something shorter like expect.schema() has it's appealing. What do you think?

[sheremet-va]

What about the naming: something shorter like expect.schema() has it's appealing. What do you think?

Maybe expect.schemaMatching to align with expect.stringMatching?

[zirkelc]

@sheremet-va I just wanted to check how we should proceed with this PR. Should I remove all other matcher except the asymmetric one, or do you want to discuss the current options with the team internally first?

Yes, please only keep the asymmetric one.

Okay, then we can keep the current version!

I added the new matcher to the docs. Let me know if there is anything else to do! 🙏

[zirkelc]

@sheremet-va the tests keep failing due to install errors. Is there something I can do?

[sheremet-va]

Can we have more failing examples that also check the generated error.diff?

[sheremet-va]

@sheremet-va the tests keep failing due to install errors. Is there something I can do?

Your pnpm-lockfile is incorrect, reinstall dependencies

[netlify]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	059fa12
🔍 Latest deploy log	https://app.netlify.com/projects/vitest-dev/deploys/68edc8846e139300085cb147
😎 Deploy Preview	https://deploy-preview-8527--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[zirkelc]

@sheremet-va the tests keep failing due to install errors. Is there something I can do?

Your pnpm-lockfile is incorrect, reinstall dependencies

I tried that multiple times, but the actual issue was that I was using pnpm v9 and package.json was set at v10. Upgrading pnpm solved it.

Can you look at the current errors, it looks like it's a general CI error. Build and test runs locally.

[sheremet-va]

Can you look at the current errors, it looks like it's a general CI error. Build and test runs locally.

The error happens because you are updating all packages instead of only installing @standard-schema/spec. Try to keep the number of changes in the lockfile to a minimum.

[zirkelc]

The error happens because you are updating all packages instead of only installing @standard-schema/spec. Try to keep the number of changes in the lockfile to a minimum.

Thanks, I wasn't aware of that - I had merge conflicts on the lockfile when pulling in upstream changes and I just deleted it and re-installed it, probably causing many lockfile changes.

I added failing tests with error.diff snapshots. The failing browser tests look unrelated, but no idea regarding the failing macos build&test.

[sheremet-va]

LGTM