Add test weight and propagate to service#264
Conversation
Introduce a weight field on TestConfig (default 100.0, non-negative) to allow tests to be weighted. Pass the configured weight into CriteriaTreeService when constructing tests so downstream logic can account for test importance.
There was a problem hiding this comment.
Pull request overview
Adds a per-test weight to the autograder configuration and threads it into criteria tree construction so downstream scoring/prioritization can account for relative test importance.
Changes:
- Added
weight(default100.0, non-negative) toTestConfig. - Updated criteria tree parsing to pass
TestConfig.weightintoTestNodeconstruction.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| autograder/models/config/test.py | Introduces the weight field on TestConfig with validation/defaults. |
| autograder/services/criteria_tree_service.py | Propagates TestConfig.weight into constructed TestNode instances. |
| test = TestNode( | ||
| config.name, | ||
| test_function, | ||
| config.get_kwargs_dict() or {}, | ||
| file_target, | ||
| config.weight, | ||
| ) |
There was a problem hiding this comment.
config.weight is now stored on TestNode, but it doesn’t appear to affect scoring because GraderService.process_test() currently creates TestResultNode without copying test.weight (it defaults to 100). This makes per-test weights effectively ignored downstream (including focus/scoring). Propagate the criteria TestNode.weight into the corresponding TestResultNode.weight (or otherwise ensure balancing/scoring uses the configured weight).
There was a problem hiding this comment.
@matheusmra does this apply?
Yes! I just commited this
| parameters: Optional[List[ParameterConfig]] = Field( | ||
| None, description="Named parameters for the test function" | ||
| ) | ||
| weight: float = Field(100.0, ge=0, description="Weight of this test") | ||
|
|
There was a problem hiding this comment.
The new weight field is part of the public config surface but there are no unit tests asserting its default (100.0) and validation (reject negative values). Please extend the existing config model tests (e.g., tests/unit/test_config_models.py) to cover these behaviors so regressions are caught.
|
@matheusmra joao was the last one to work on this and as far as i remember tests already had weights. @jaoppb can you take a look here? |
|
@matheusmra nevermind! Just read the issue description. |
ArthurCRodrigues
deleted the
263-bug-penaltyweight-field-lost-during-configuration-retrieval
branch
This pull request introduces support for assigning a weight to each test in the autograder configuration, allowing for more granular scoring or prioritization of tests. The main changes add a
weightfield to theTestConfigmodel and ensure this value is passed through to where tests are constructed.Test configuration and scoring improvements:
weightfield (defaulting to 100.0, must be non-negative) to theTestConfigmodel inautograder/models/config/test.py, allowing each test to specify its relative importance.autograder/services/criteria_tree_service.pyto pass the newweightfield when constructing a test node, ensuring the weight is used in downstream processing.