Benchmarks are measurements, not the product

MMLU measures multiple-choice accuracy across 57 subjects. HumanEval measures functional correctness for 164 handwritten Python problems. SWE-bench measures whether a system can modify repositories to resolve real issues. AgentBench measures task completion across eight interactive environments. Their units of work range from one answer to an extended model-tool trajectory.

The original SWE-bench collected 2,294 instances from issues and pull requests across 12 Python repositories. OpenAI and the benchmark authors later created SWE-bench Verified, a human-reviewed subset of 500 instances, after finding examples with underspecified issue text, overly specific tests, and environment problems.

The Verified work also highlights a system-level fact: the same model can produce different results under different agent scaffolds. Repository browsing, tool interfaces, context construction, editing strategy, and retry policy are part of the evaluated system.

The SWE-bench project now publishes a bash-only comparison that holds the mini-SWE-agent environment more constant across models. Its documentation still warns that results from major scaffold releases are not necessarily comparable. Standardizing more of the system improves attribution, but it does not remove dataset limitations.

Accuracy counts one selected answer. Pass@k estimates whether at least one of k samples is correct. Repository resolution requires all designated tests to pass. Agent environments define their own success conditions. Each metric rewards a different operating procedure.

A metric can reject a useful result when the reference is too narrow, or accept an incomplete result when tests have low coverage. Human review can catch some of these problems, but reviewers also need a rubric, representative expertise, and disagreement handling.

Report uncertainty around the measurement. Repeated sampling, task subsampling, grader variation, and environment instability can change results. A small score difference is not automatically meaningful if it is smaller than normal run-to-run or review variation.

A public benchmark is reproducible because its tasks and often its solutions are available. That same openness creates contamination risk when training corpora include benchmark pages, repositories, discussions, or solution artifacts. Performance may then reflect prior exposure as well as general problem-solving ability.

In a 2026 analysis, OpenAI reported evidence that frontier models could reproduce details from SWE-bench Verified problems or gold patches and stopped using that benchmark to measure frontier coding progress. The conclusion is specific to that measurement regime, but the evaluation principle is general: track exposure risk and refresh tasks when the signal degrades.

Private or newly created holdout tasks reduce direct exposure, but they still require quality review. The strongest program combines controlled holdouts with transparent public benchmarks, real product tasks, and detailed failure analysis rather than relying on one secret number.

Start with real tasks sampled from the product's intended use. Freeze the input state, define acceptable outcomes and prohibited changes, and run the same task across candidate systems. Keep a held-out set for release decisions and a diagnostic set for rapid iteration.

For coding agents, combine automated checks with human review. Tests can establish behavior on covered cases; reviewers can judge scope control, maintainability, explanation quality, and whether the agent made unsafe or unnecessary changes.

Static public benchmarks can become less discriminating as models and training corpora evolve. Product tasks also drift when users, tools, and policies change. Version datasets, record failures, add new cases from incidents, and preserve old cases that protect important behavior.

Publish disaggregated results. An average can hide a critical regression in one task family or user group. A useful evaluation report says what was tested, where the system improved, where it failed, and what remains outside the scope of the evidence.

A July 2026 audit of SWE-Bench Pro estimated that roughly 30 percent of its public tasks were broken, with problems including strict tests, underspecified prompts, low-coverage tests, and misleading instructions. The authors retracted an earlier recommendation to use that benchmark. This is a useful example of evaluation governance working: inspect the data, publish the failure, and change the claim.