Literature eval enhancements#28

Open

valerie-autumn-skye wants to merge 34 commits intomainfrom

literature-eval-enhancements

Commits on Aug 29, 2025

Enforce UTF-8 for Goose session files.
Charles Parker
committed
Fixes issue #15. Prevents divide by zero errors and cleans up summaries by using consistent printing methods.
Charles Parker
committed
Cleaned up output by using consistent printing methods.
Charles Parker
committed

Commits on Aug 30, 2025

Fixes Issue #18 by implementing metric downgrades to Claude if OpenAPI calls fail, and to DummyMetric if Claude fails.
Charles Parker
committed
Satisfied ruff's bizarre rules.
Charles Parker
committed
Added extra logging and test for goose UTF-8 handling.
Charles Parker
committed
Added metacoder configuration test cases for claude downgrade and no server combinations to support Issues #18, #19, and #20.
Charles Parker
committed
Added unit test for claude downgrade to support Issue #18. Cleaned up logging in runner.py. Added test configuration to support log capture for assertions that downgrade was successful.
Charles Parker
committed
Added unit test for claude downgrade to support Issue #18. Cleaned up logging in runner.py. Added test configuration to support log capture for assertions that downgrade was successful. Addressed r…
Charles Parker
committed
Added assertion to confirm that ClaudeJudge completed scoring the metric after the downgrade.
Charles Parker
committed
Added assertion to force test to fail on Exception. Increased logging verbosity temporarily to debug Claude judge unit test on build server. Adjusted logic to work when multiple coders are specifie…
Charles Parker
committed
Fixed runtime issues related to metric downgrade from CorrectnessMetric to DummyMetric.
Charles Parker
committed
Added test coverage of new evaluation judge functionality. Added test for the quota exhaustion fallback logic.
Charles Parker
committed

Commits on Sep 2, 2025

Commits on Sep 3, 2025

Commits on Sep 4, 2025

Added UTF-8 encoding to prevent character mangling during YAML export on Windows (where the default codepage is cp1252).
Charles Parker
committed
Added support for grouping test case eval results with 'group' key in config. Fixes Issue #27.
Charles Parker
committed
Updated test_runner.py to include Default case_group in EvalResults to address validation errors in test suite.
Charles Parker
committed
Updated Anthropic fallback mode from claude-3-5-sonnet-20240620 to claude-sonnet-4-20250514.
Charles Parker
committed
Corrected test cases to match the expected Anthropic model.
Charles Parker
committed

Commits on Sep 5, 2025

Commits on Sep 18, 2025

Refactored OS environment detection to create relative paths for coder configs consistently.
Charles Parker
committed