Integrate pytest-subtests #13738

nicoddemus · 2025-09-20T11:59:58Z

This PR copies the files from pytest-subtests and performs minimal integration.
I'm opening this to gauge whether everyone is on board with integrating this feature into the core.

Why?

Pros

subtests is a standard unittest feature, so it makes sense for pytest to support it as well.
Provides a simple alternative to parametrization.
Adds the ability to generate new test cases at runtime during the test execution, which is not possible with parametrization.
While it can exist as an external plugin, it requires many hacks, and better report integration is not easily achievable without core integration (off the top of my head: issues with terminal reporting, last failed and stepwise support, among others).

Cons

Adds another maintenance burden to the core.

TODO ✅

If everyone is on board, I will take the time this week to polish it and get it ready to merge ASAP:

✅ Cleanup the implementation: Currently it relies on monkey-patching, which should no longer be needed since we can modify the core code directly.
✅ Documentation: The feature should be documented as experimental -- meaning that while the feature itself is solid, we are still working out details such as how to best report subtest failures, integration with other plugins, etc.
✅ Fix lint and testing failures (obviously).

Consider spelling this Subtests. This way it matches the fixture name subtests (rather than sub_tests). Generally the sub- prefix is not followed by a capital letter AFAIK. I understand it's inconsistent with unittest spelling subTest but that's on them :)

Note I don't feel strongly about this, if you prefer SubTests that's fine.

bluetech · 2025-09-30T07:49:31Z

src/_pytest/subtests.py

+
+# Note: cannot use a dataclass here because Sphinx insists on showing up the __init__ method in the documentation,
+# even if we explicitly use :exclude-members: __init__.
+class SubTests:


Another naming point, for some fixture types we use the Fixture suffix, e.g. CaptureFixture. But for some we don't (e.g. MonkeyPatch, Cache, Pytester). Perhaps the difference is whether the class only makes sense in a fixture context (like CaptureFixture) or if it's usable independently. For SubTests I believe it's strictly a fixture, so that would suggest SubTestsFixture. But that's a bit longer.

I wish we were consistent on this. I'll leave the decision to your good judgment.

bluetech · 2025-09-30T07:51:15Z

src/_pytest/subtests.py

+class SubTests:
+    """Subtests fixture, enables declaring subtests inside test functions via the :meth:`test` method."""
+
+    def __init__(


Add _ispytest privacy check?

bluetech · 2025-09-30T08:21:44Z

doc/en/how-to/subtests.rst

+
+pytest allows for grouping assertions within a normal test, known as *subtests*.
+
+Subtests are an alternative to parametrization, particularly useful when test setup is expensive or when the exact parametrization values are not known at collection time.


For expensive test setup, pytest has a solution in the form of scoped fixtures. Since regular parametrization is advantageous, e.g. allowing each param value to be tested individually by nodeid, I think we should not encourage subtests when there's a decent parametrization alternative.

The "values not known at collection time" case is what I think we should emphasize. There is indeed inherently no way to do "dynamic parametrization" after collection.

bluetech · 2025-09-30T08:24:56Z

testing/test_subtests.py

+        )
+
+
+class TestSubTest:


Suggested change

class TestSubTest:

class TestUnittestSubTest:

bluetech · 2025-09-30T08:25:27Z

testing/test_subtests.py

+
+
+class TestSubTest:
+    """Test.subTest functionality."""


Suggested change

"""Test.subTest functionality."""

"""Test subTest() functionality in unittest tests."""

Unless I misunderstood.

bluetech · 2025-09-30T08:28:22Z

testing/test_subtests.py

+
+
+@pytest.mark.parametrize("mode", ["normal", "xdist"])
+class TestFixture:


Would be nice to have a test for the combination of parametrization + subtests. I'm sure it works, but it's the sort of thing that I can see might break if a different implementation strategy was used.

bluetech · 2025-09-30T08:35:35Z

doc/en/how-to/subtests.rst

+---------------------------
+
+While :ref:`traditional pytest parametrization <parametrize>` and ``subtests`` are similar, they have important differences and use cases.
+


Continuing from the comment above, I can see two ways we can approach this:

1 - Subtests are for "runtime parametrization"

Per comment above, subtests are useful when you have some data dynamically fetched in the test and want individualized reporting for each data value.

The idea here is that parametrization should be the go-to tool, but we offer this subtest tool for this particular scenario.

2 - Subtests are for "sub-testing"

By which I mean, subtests are for when you have one conceptual test, i.e. consider it a complete whole, but just want to break down its reporting to parts.

How do you see it? The reason I'm asking is that it can affect how we document the feature, what we recommend, etc.

nicoddemus changed the title ~~[DRAFT] Integrate pytest subtests~~ [DRAFT] Integrate pytest-subtests Sep 20, 2025

nicoddemus commented Sep 20, 2025

View reviewed changes

src/_pytest/subtests.py Outdated Show resolved Hide resolved

nicoddemus force-pushed the integrate-pytest-subtests branch from b43ab38 to 97ee032 Compare September 22, 2025 23:07

nicoddemus mentioned this pull request Sep 23, 2025

minor: do not import pytest in unittest.py #13748

Merged

nicoddemus force-pushed the integrate-pytest-subtests branch from 97ee032 to 6b5831f Compare September 26, 2025 13:34

psf-chronographer bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Sep 26, 2025

nicoddemus force-pushed the integrate-pytest-subtests branch 2 times, most recently from 5f56d81 to c93c0e0 Compare September 26, 2025 14:00

nicoddemus marked this pull request as ready for review September 26, 2025 14:00

nicoddemus changed the title ~~[DRAFT] Integrate pytest-subtests~~ Integrate pytest-subtests Sep 26, 2025

nicoddemus added 6 commits September 29, 2025 19:17

Add pytest-subtests files changes

c3dc9f4

In addition, enable the plugin in `pytest/__init__.py` and `config/__init__.py`.

subtests: remove direct pytest import

e856b0b

Force using xdist plugin and fix linting

596a4dc

Replace attr by dataclass

8f58851

Add docs

40b6cb2

Cleanup internal hacks

b569c93

nicoddemus force-pushed the integrate-pytest-subtests branch from c93c0e0 to b569c93 Compare September 29, 2025 22:17

bluetech reviewed Sep 30, 2025

View reviewed changes

		from _pytest.runner import check_interactive_exception


		if TYPE_CHECKING:


		pytest allows for grouping assertions within a normal test, known as subtests.

		Subtests are an alternative to parametrization, particularly useful when test setup is expensive or when the exact parametrization values are not known at collection time.

	"""Test.subTest functionality."""
	"""Test subTest() functionality in unittest tests."""



		@pytest.mark.parametrize("mode", ["normal", "xdist"])
		class TestFixture:

		---------------------------

		While :ref:`traditional pytest parametrization <parametrize>` and ``subtests`` are similar, they have important differences and use cases.

		)


		@dataclasses.dataclass

Uh oh!

Integrate pytest-subtests #13738

Are you sure you want to change the base?

Integrate pytest-subtests #13738

Uh oh!

Conversation

nicoddemus commented Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why?

Pros

Cons

TODO ✅

Related

Uh oh!

Uh oh!

Pierre-Sassoulas commented Sep 20, 2025

Uh oh!

RonnyPfannschmidt commented Sep 20, 2025

Uh oh!

RonnyPfannschmidt commented Sep 20, 2025

Uh oh!

webknjaz commented Sep 20, 2025

Uh oh!

nicoddemus commented Sep 26, 2025

Uh oh!

bluetech left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

1 - Subtests are for "runtime parametrization"

2 - Subtests are for "sub-testing"

Uh oh!

Uh oh!

nicoddemus commented Sep 20, 2025 •

edited

Loading