ensure-test-coverage

Run the autolint tool with uv run python tools/run_autolint.py <eval_name> for structural checks, then run pytest with --cov=src/inspect_evals/<eval_name> --cov-report=term-missing --runslow for line-level coverage. This Skill automates both steps and interprets the results.

Build a TaskState with a mockllm model, set state.output with ModelOutput.from_content, call the scorer with a Target, and assert the score value is CORRECT or INCORRECT. Parametrize inputs instead of writing near-duplicate test functions.

No. Solvers that only assemble Inspect built-ins like react() or chain() need at most an isinstance type check, and reviewers explicitly reject deeper tests of their internals. Only solvers with custom branching logic require behavioral tests.

Use the shared utilities in tests/utils/sandbox_tools.py: create_sandbox_tool_task builds a minimal Task with your compose.yaml, and assert_sandbox_test_passed verifies the result. Mark these tests with @pytest.mark.docker and @pytest.mark.slow with the observed duration.

Do not use it to fix a specific failing CI test; that belongs to a CI maintenance workflow. It is designed for reviewing or creating an evaluation's overall test suite, not debugging individual test failures.

Tests using actual records from the dataset serve as both verification and documentation of the data schema. Fabricated examples can drift from the real dataset structure and miss parsing bugs that real records would expose.