Start with the task distribution
Model selection begins with the work users actually perform. A small code transformation, a repository-wide migration, a factual research task, and an interactive tool workflow stress different capabilities. One public score cannot represent all of them.
Write down the input shape, allowed tools, expected output, cost of a wrong answer, and how correctness will be checked. Then construct an evaluation set from representative tasks, including ordinary cases, difficult edge cases, and failures that previously reached users.
The task distribution should include frequency and consequence. Common low-risk edits determine everyday responsiveness, while rare high-impact actions determine whether the system needs stricter routing or approval. Sampling only impressive demonstrations will select a model for a product that users do not actually have.
A task-first model selection funnel
Selection begins with product work and ends with a verified operating policy. Public benchmarks inform the middle, but they do not define the product requirement.
Observed work
Collect representative tasks
Sample routine requests, costly edge cases, and known failure patterns from the intended workflow.
Success contract
Define checks and boundaries
Specify acceptable outputs, prohibited changes, tools, budgets, and the evidence required for completion.
Candidate systems
Run comparable protocols
Use the same initial state, scaffold, prompt policy, and evaluation procedure for every candidate.
Independent measures
Compare quality and cost
Review task success, reliability, tool behavior, latency, safety, and human correction effort separately.
Operating policy
Select, route, and monitor
Choose the simplest policy that meets requirements, then watch production failures and distribution drift.
Evaluation approach aligned with HELM's emphasis on scenarios, metrics, standardization, and explicit incompleteness.
Compare independent dimensions
HELM was created to make model evaluation broader, multi-metric, and standardized. Its framework emphasizes that evaluation is unavoidably incomplete and should cover multiple scenarios and metrics rather than collapse everything into one notion of accuracy.
Keep dimensions disaggregated until the decision requires a tradeoff. A weighted average can be useful for routing, but it can also hide a safety failure behind gains in speed or hide a severe regression inside a large category. Report both the decision score and its underlying components.
- Quality: Does the output satisfy the task and its hidden edge cases?
- Reliability: How much does performance vary across repeated runs and prompt forms?
- Tool behavior: Does the system choose, call, and recover from tools correctly?
- Efficiency: What latency, token use, and external-tool cost produce a verified result?
- Safety: Does the model respect authority, data boundaries, and high-impact approvals?
Source figure
HELM visualized the coverage problem in model evaluation
Figure 4 from HELM contrasts uneven prior evaluation coverage with a standardized matrix of models and common scenarios. Blank cells are missing evidence, not zero capability.
Liang et al., Figure 4| Task shape | Primary requirement | Useful evaluation | Typical control |
|---|---|---|---|
| Bounded transformation | Instruction fidelity and low latency | Exact fixtures plus review of unnecessary changes | Limit editable scope |
| Factual research | Evidence quality and calibrated claims | Source recall, citation support, and abstention cases | Require provenance |
| Repository repair | Long-horizon tool use and regression control | Repository tasks with tests and diff review | Sandbox and verify writes |
| High-impact action | Policy compliance and predictable refusal | Adversarial permission and approval scenarios | Deterministic authorization |
Use benchmarks as lenses
MMLU contains 57 multiple-choice tasks spanning academic and professional subjects. HumanEval contains 164 handwritten Python programming problems evaluated through functional tests. SWE-bench instead asks systems to resolve real GitHub issues in repository environments. These benchmarks measure different units of capability, so their scores are not interchangeable.
Match the lens to the product. A coding assistant that edits repositories needs repository-level tasks, tool use, regression checks, and review quality. A multiple-choice knowledge benchmark can add context, but it cannot substitute for that workflow evaluation.
Benchmark protocol details also affect interpretation. HumanEval's pass@k asks whether at least one of k generated samples is functionally correct, so pass@1 and pass@100 answer different operational questions. A product that presents one answer cannot assume the benefit of generating and testing a large private sample set unless it actually performs that selection.
| Benchmark | Published protocol | Reported result | What the value means |
|---|---|---|---|
| HumanEval | Codex 12B, one generated sample per problem | 28.81% pass@1 | A passing Python function among one sample |
| HumanEval | Codex 12B, 100 samples per problem | 72.31% pass@100 | At least one passing function among 100 samples |
| SWE-bench | Claude 2 in the original paper setup | 1.96% resolved | Repository issues whose designated tests passed |
| HELM | 30 models evaluated on 16 core scenarios | 96.0% scenario coverage | Coverage of a common evaluation matrix, not accuracy |
Measure the cost of reaching a reviewable result
A model output is not yet a product outcome. Include retries, tool calls, tokens, wall-clock time, human review, and corrective turns when calculating cost. A cheap first response can become expensive if a developer must discover and repair hidden mistakes.
Run repeated trials for tasks where sampling changes behavior. Report variance and failure clusters rather than only the mean. If one candidate succeeds most of the time but occasionally violates scope, that tail behavior may dominate the deployment decision.
Use blind review when practical. Reviewers should see the artifact and task requirements without being influenced by the model brand. Record why a result failed, because labels such as wrong answer, missing evidence, excessive edit, unsafe action, and poor recovery lead to different improvements.
Route only when the policy is explainable
A product may route simple requests to a faster model and complex or high-risk tasks to a more capable one. The routing rule should be testable, visible in logs, and evaluated with the same task set as the models it selects.
Measure time to a verified result, not only model response time. A slower model can be cheaper overall if it reduces retries and review effort; a faster model can be better when feedback speed matters and errors are easy to detect. The correct choice is a property of the workflow, not the model name alone.
Fallbacks need their own evaluation. If the preferred model is unavailable, the system may reduce tool access, ask the user to wait, or route to another model. Silent substitution is risky when the replacement has not passed the same safety and task requirements.
Sources and further reading
Claims and figures in this article were checked against these original papers and official project resources.
- 1Holistic Evaluation of Language Models
Stanford Center for Research on Foundation Models, 2022
- 2Measuring Massive Multitask Language Understanding
International Conference on Learning Representations, 2021
- 3Evaluating Large Language Models Trained on Code
OpenAI, 2021
- 4SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
International Conference on Learning Representations, 2024
