Which model is best? On paper, it sounds like a straightforward question. Every provider launch features a leaderboard, every leaderboard crowns a winner, and delivery dates are routinely staked on whichever model tops the charts that fortnight.
In practice, the moment you have to build on a model and bet a deadline on it, that neat ranking dissolves. A score that looks decisive on a vendor’s presentation slide tells you almost nothing about how the model will hold up when deployed to a client.
We tackled this at a recent Enablers Circle — a bi-weekly session where our engineering team meets to hash out technical and industry challenges. We started with a deceptively simple constraint: if you could keep only three model metrics to bet a delivery date on, which would they be?
By the end of the session, we reached a consensus: standard public benchmarks rarely capture what actually matters in production.
The Anatomy of a Benchmark
At its core, an LLM benchmark is a standardised test. You run a model through a fixed set of tasks and reduce performance to a single comparable score. They are ubiquitous because they offer a quick answer to a complex comparison:
- SWE-bench measures agentic coding against real GitHub issues.
- MMLU tests broad domain knowledge.
- LMArena ranks models using double-blind human feedback.
The industry, however, is growing increasingly sceptical of these scores. Top-tier frontier models are starting to saturate standard benchmarks, and test questions frequently leak into training datasets — a problem known as contamination.
Which Metrics Actually Matter?
When you put a list of candidate metrics on the wall and ask engineers to choose three, the room splits immediately:
- Codingwriting and fixing code
- Instruction followingdoes it do what you tell it
- Tool usecalling external tools like file editors and APIs, not just chatting
- Agentic orchestrationcoordinating multi-step work across sub-agents
- Hallucination ratehow often it states plausible falsehoods
- Cost per tokenthe price of every unit of text going in and out
- Long contexthow much material it can hold in mind at once
- Human preferencewhich model people simply like using
Coding and instruction following are usually the first picks, but pushback comes quickly. High-end models have made routine code completion reliable enough that it feels baseline. It only seems secondary because it works; drop back to 2022-era completion and you notice instantly.
Tool use provokes a similar debate. Two years ago it was a secondary feature; today, for many integration patterns, it is foundational. Strip away reliable function calling, and an enterprise system degrades back into a basic chatbot.
Conversely, engineers working on infrastructure and platform architecture care less about line-by-line code generation and more about multi-step reasoning.
The takeaway from the session:
A metric’s importance is defined by the specific workload, not as an intrinsic property of the model itself.
Real-World Constraints Shift the Priority
To test this, we evaluated metric selection across three distinct client scenarios:
1. Regulated Claims Triage (Insurance)
Priority: Low Hallucination Rate & Deterministic Control.
Context: In heavily audited environments, confidence and accuracy override raw capability. A model making up facts creates immediate regulatory risk. Teams need predictable execution within strict guardrails, ideally hosted within private network boundaries.
2. High-Volume Retail Support
Priority: Cost per Token & Latency.
Context: When serving millions of customer interactions, unit economics dominate. If a bot is restricted to retrieving answers from a curated product catalogue, raw intelligence matters less than cost efficiency and response time at scale.
3. Legacy Codebase Migration
Priority: Context Retention vs. Error Compounding.
Context: Migrating massive, poorly documented legacy codebases seems like a test for massive context windows. However, feeding huge chunks of unstructured code into a model often increases hallucination rates. Because nobody evaluates thousands of generated lines manually, minor errors compound across the migration pipeline.
The broader realisation was that no team should attempt a “one-shot” migration of a massive application anyway. Good architecture breaks the work down into modular tasks — making ultra-long context windows less critical than structured breakdown and validation.
The Problem with Public Leaderboards
Why do public scores fail to translate to production reliability?
- Synthetic conditions: Benchmarks run in sterile environments. Real codebases have technical debt, messy schemas, ambiguous requirements, and complex deployment pipelines.
- Data contamination & overfitting: Vendors optimise models against public test sets. A high score often reflects rote memorisation of the test rather than generalised reasoning.
- Selective reporting: Provider launch slides feature benchmarks where their model wins and omit those where it underperforms.
A single metric like 88.4% on SWE-bench offers no insight into how a model handles your specific domain, architectural patterns, or data sensitivity.
Personal Tools vs. Production Systems
The debate resolved when we separated model selection into two distinct categories:
1. Developer Tooling & Personal Productivity
When evaluating models for personal use (IDEs, CLI tools, draft generation), human preference and convenience win. Benchmarking slight variations between top-tier models yields diminishing returns. For day-to-day developer workflows, using modern models and staying near the technical frontier is sufficient.
2. Embedded Production Architecture
When a model is integrated into a customer-facing or mission-critical system, you cannot simply swap versions as new leaderboards update. Production implementations require:
- Pinned model versions: Preventing unexpected behaviour shifts from subtle upstream updates.
- Custom evaluation suites: Running a representative suite of domain-specific tasks against your non-functional requirements (NFRs) before any deployment.
- Continuous monitoring: Tracking output drift, accuracy decay, and cost changes over time.
Conclusion
There is no single benchmark score that guarantees a model will perform in your environment. Public leaderboards provide a general signal about overall industry movement, but they shouldn’t drive architectural decisions.
Engineering teams should:
- Define what “success” means for their specific workload before comparing models.
- Use intuitive tools and modern frontier models for personal productivity without over-analysing minor differences.
- Build custom, domain-specific evaluation pipelines for production applications to measure latency, cost, and reliability against real criteria.
Ultimately, model selection isn’t about finding the highest score on a vendor slide — it’s about ensuring predictable execution inside your specific system.
