`ResultCollector` exception handling: improvement for `pytest.raises()` failures

[edoardob90]

Problem Description

This issue has been observed in practice when testing the 02_control_flow.ipynb notebook, but cannot be reliably reproduced in current testing environments. When a student's solution function failed to raise an expected exception in a pytest.raises() context, the test was incorrectly classified as TestOutcome.TEST_ERROR instead of TestOutcome.FAIL, causing the notebook to display a misleading "🚨 Syntax Error" message instead of "❌ Failed".

Current Status

Reproduction: Intermittent/unreproducible - the issue was observed once but cannot be triggered consistently.

Likely cause: The current code theoretically should work because pytest.fail.Exception and _pytest.outcomes.Failed are the same class object (pytest aliases them). However, the intermittent nature suggests there may be edge cases related to pytest version differences, hook execution order, or specific test scenarios.

Analysis

In tutorial/tests/testsuite/helpers.py at lines 759-778, the ResultCollector.pytest_exception_interact() method catches AssertionError and pytest.fail.Exception, but does NOT explicitly reference _pytest.outcomes.Failed, which is the canonical name for the exception raised when pytest.raises() fails.

python-tutorial/tutorial/tests/testsuite/helpers.py

Lines 767 to 772 in 047f9b4

	outcome = (
	TestOutcome.FAIL
	if exc.errisinstance(AssertionError)
	or exc.errisinstance(pytest.fail.Exception)
	else TestOutcome.TEST_ERROR
	)

Why is it happening?

When a test uses pytest.raises() but the student's solution doesn't raise the expected exception:

1. Test with pytest.raises(): Student's solution should raise ValueError, but doesn't
2. pytest behavior: pytest.raises() context manager's __exit__ method detects no exception was raised
3. pytest raises Failed: Internally raises _pytest.outcomes.Failed with message like "DID NOT RAISE ValueError"
4. Exception handling: pytest_exception_interact() is called but doesn't recognize Failed exception
5. Misclassification: Defaults to TestOutcome.TEST_ERROR
6. Escalation: run_pytest_for_function() (in testsuite.py) sees TEST_ERROR and returns IPytestOutcome.PYTEST_ERROR
7. Display: Notebook shows "🚨 Syntax Error" (orange) instead of "❌ Failed" (red)

Impact

Affected tests:

• Any test using pytest.raises(SomeException) where the student's code fails to raise that exception

Examples:

• test_02_control_flow.py:492 - test_base_converter_invalid_bases
• test_12_functions_advanced.py:213,231 - test_once_twice
• test_magic_example.py:55 - test_power2_raise

Students receive confusing feedback suggesting a syntax/compile error rather than understanding they failed to raise the expected exception.

Proposed Hotfix

Even though pytest.fail.Exception and _pytest.outcomes.Failed are technically the same class object in pytest's current implementation, we recommend explicitly referencing _pytest.outcomes.Failed for the following reasons:

1. Code clarity: Makes it explicit that we're handling the canonical exception from pytest.raises() failures
2. Documentation: Self-documents that this code path handles failed pytest.raises() assertions
3. Defensive programming: Guards against potential pytest version differences or internal refactoring
4. Edge case protection: May prevent issues related to import order, hook execution timing, or environment-specific behaviors
5. Direct import: Uses the actual exception class rather than relying on the .Exception attribute pattern

Changes

In tutorial/tests/testsuite/helpers.py:

• Add import somewhere (preferably in the function's scope):

from _pytest.outcomes import Failed

• Update lines 767-772:

outcome = (
    TestOutcome.FAIL
    if isinstance(exc.value, AssertionError | Failed)
    else TestOutcome.TEST_ERROR
)

This explicitly catches all pytest assertion failures (direct assertions, pytest.fail(), and pytest.raises() failures) as FAIL, while preserving TEST_ERROR classification for actual syntax errors, import errors, etc.

Technical Note

In pytest's current implementation, pytest.fail.Exception and _pytest.outcomes.Failed reference the same class:

• _pytest.outcomes.Failed is the actual exception class
• pytest.fail.Exception is created by the @_with_exception decorator that attaches the exception class to the fail() function
• pytest.raises.Exception is just an alias of fail.Exception

Testing Considerations

Test scenario to verify behavior:

Use existing tests like test_02_control_flow.py::test_base_converter_invalid_bases:

# In notebook cell (02_control_flow.ipynb):
%%ipytest
def solution_base_converter(number, from_base, to_base):
    # Intentionally doesn't raise ValueError for invalid bases
    return "42"

Expected behavior:

• With the fix, the test should consistently show "❌ Failed" with a clear message about not raising the expected ValueError
• The defensive fix ensures this behavior is stable across different pytest versions and environments