Every enterprise LLM system will hit a moment where the team argues about whether the model got better. If nobody in the room can produce a number, the argument is not resolvable and the system is not maintainable.
The most consequential AI decision most enterprise engineering leaders will make in 2026 is not which foundation model to standardize on. It is whether their team can produce, on demand, a defensible number that answers the question "did the system get better or worse this week." The teams that can produce that number will iterate the system to genuine business utility. The teams that can't will iterate it into a status meeting where the product manager says the users seem happier and the engineering lead says the eval scores are hard to compare across releases and the executive sponsor asks whether anyone is actually measuring this.
The industry has a name for the discipline that produces the number. It is called evaluation, or evals, and it is the closest thing modern AI engineering has to a real professional practice. It is also the discipline most enterprise teams are shortest on, most vendors are quietest about, and most executive teams underestimate the cost of building.
The public work on evals has gotten very good in a short time. Stanford's HELM benchmark suite has spent five years building open, holistic evaluation frameworks that compare models across a broad set of scenarios and metrics. Anthropic maintains a public evals repository with concrete methodologies for measuring model behavior under specific conditions. OpenAI's evals framework, open-sourced since 2023, provides a template for how a serious evaluation pipeline is structured. LMSYS's Chatbot Arena has run millions of human-preference comparisons and published the methodology behind them. The academic literature on LLM evaluation now runs into thousands of papers per year.
None of this public work substitutes for the evaluation your team needs to build for your system. It informs that work — it shows you the shape of the discipline, the vocabulary, the failure modes to design against — but the eval that matters for your production system is the one that measures your system, on your data, against the outcomes your users are trying to achieve. Nobody else can build that eval for you, and no vendor benchmark will substitute for it. This is the part that trips up executive teams: they see the impressive public benchmarks, assume "the model is already evaluated," and skip the step of building the evaluation that is actually load-bearing for their operation.
What "the evals are the product" actually means
The phrase circulates in AI-engineering circles as shorthand. It compresses three arguments that, taken together, reorient how a team should think about building on top of foundation models.
Argument one: model choice is roughly commoditized within a capability tier. The frontier models from the top three or four providers are close enough on most enterprise-relevant benchmarks that switching between them, within a capability tier, is a smaller performance change than most teams believe. What differentiates a working system from a broken one is not the model. It is the retrieval strategy, the prompt architecture, the tool interfaces, the fallback logic, the safety net around the output, and the operational discipline of the team maintaining it. All of those live in the eval framework.
Argument two: model releases will keep happening, and your system has to survive them. Every foundation model provider ships new versions on a rolling basis. Some of them improve your workflow's performance. Some of them regress it, subtly, on a subset of inputs that only matters to you. Without a repeatable eval, you cannot tell which is which. With one, you can adopt new model versions on your own terms — running the new model against the eval, comparing the delta, deciding whether to switch. Without one, you either freeze the old model forever (losing the improvements) or upgrade blindly (accepting the regressions). Both are worse than being able to make an informed switch.
Argument three: the eval is the closest thing your team has to a specification. Every LLM system is, in a certain sense, undefined. The model's behavior is not fully described by its documentation. The system's behavior is not fully described by the code. What the system is supposed to do — the specification, in the classical software-engineering sense — lives in the eval. If the eval says "on this input, the correct output is X, and outputs Y and Z are wrong," then the system's behavior on that input is specified. If there is no eval, there is no specification. There is only vibes, and the vibes are held in the heads of two or three engineers who won't be on the team in eighteen months.
What a working eval framework looks like
The shape of a serious eval framework is now well understood. It has four components, each of which requires unglamorous work.
A golden dataset. A curated set of inputs representative of production traffic, each with a documented expected output (or a set of acceptable outputs, or a rubric for grading), maintained under version control, sized somewhere between a few hundred and a few thousand examples depending on the diversity of the workflow. The golden dataset is the load-bearing artifact of the whole framework. Building it is expensive — it requires domain-expert time, because the domain expert is the one who knows what "correct" means — and maintaining it requires ongoing discipline as the workflow evolves. Teams that skip this step end up evaluating against test cases their engineers made up, which measures whether the model can pass their engineers' guesses, not whether it can serve real users.
Offline metrics. Automated scoring against the golden dataset, run on every model change, on every prompt change, on every retrieval-strategy change. The specific metrics depend on the workflow — accuracy, F1, ROUGE, semantic similarity, LLM-as-judge scores, factuality flags — and the choice matters more than most teams realize. A metric that doesn't correlate with the business outcome the system is meant to produce is worse than no metric; it produces false confidence. The "LLM-as-judge" methodology from the Zheng et al. paper is now the default choice for open-ended outputs, and understanding its failure modes — position bias, verbosity bias, self-preference — is table stakes for any team using it.
Online metrics. Continuous measurement of a sample of production traffic, scored against the same rubric as offline. This is where teams discover that their offline evaluation was measuring the wrong thing. Production traffic doesn't look like the golden dataset. It doesn't look like anyone's guess about what production traffic will look like. It looks like what real users actually type, which is stranger, longer, more ambiguous, and more error-prone than any test set. The online metric is the corrective. It also produces the feedback loop that keeps the golden dataset current — real production failures get added to the golden set, and the next model iteration is evaluated against a dataset that reflects what users actually do.
A specific human owner. The eval framework is a product. It has an owner. That owner's job is to keep the golden dataset current, to run the eval before every deployment, to publish the results, and to raise the flag when the numbers move. In smaller teams this is a rotating responsibility; in larger teams it is a dedicated role that reports to engineering leadership. What it cannot be is "everybody's job," because everybody's job is nobody's job, and eval frameworks that don't have a named owner rot in a Colab notebook within six weeks.
Why enterprises get this wrong
The reason enterprise teams under-invest in evaluations is not that the discipline is hard to learn. The public literature and the open-source tooling are more than sufficient. The reason is that eval work is deeply unglamorous, expensive to build, and invisible in the wins column until something goes wrong. It is engineering hygiene, in the same category as observability, load testing, and CI/CD — the disciplines that no product manager gets promoted for, and that every senior engineering leader has watched a team ship without, until the moment they wish they hadn't.
The specific failure sequence, in the LLM context, is worth naming. A team ships an LLM feature. The feature works. The team congratulates itself. Six weeks later, the model provider ships a new version. The engineering lead updates the model reference in the config. Nothing obvious breaks. Two weeks after that, a customer support ticket comes in about the feature "getting worse." The team investigates and cannot reproduce. Two weeks after that, three more tickets come in, and it is now clear that the feature is degraded for a specific class of query the golden — well, there was no golden — the test suite didn't cover. The team rolls back the model. It is now four weeks behind, running an increasingly-stale version of the foundation model. They have committed to a manual eval pass before any future model change. They will not do this because manual eval passes take days and the team has other priorities. In six months, they will ship a rewrite on a different model provider, and repeat the cycle.
This is not hypothetical. It is the most common failure trajectory I have seen in enterprise LLM programs over the last two years. It is preventable. The prevention is expensive. The prevention pays back on the second or third avoided outage, and continues paying back for as long as the system runs.
The vendor incentive problem
Foundation model providers have a genuinely tricky incentive around evaluations. On one hand, they benefit from customers running rigorous evals: it produces the data they need to iterate their models, it surfaces failure modes their internal red-teams miss, and it builds the operational maturity that lets enterprises deploy their models at scale. On the other hand, an enterprise customer with a rigorous eval framework is a customer who can accurately compare provider A's model to provider B's model on the specific workflow that matters, and switch on the basis of the comparison. That is a customer whose loyalty is difficult to buy through marketing.
Provider behavior reflects this tension. Every major model provider publishes some public evaluation work; none of them make it as easy as they could for a customer to build their own private evals. Anthropic's documentation on prompt caching and tool use is thorough; the eval tooling is less prominent. OpenAI open-sourced its evals framework early but the enterprise-facing tooling has been slower to mature. Google's Gemini evaluation surface exists but is deeply integrated with Vertex AI. The picture, across providers, is one where evaluations are visibly on the roadmap but not prioritized above features that drive short-term consumption.
The read for enterprise buyers is simple. Do not expect the model providers to build your eval framework for you. Do not accept vendor benchmarks as substitutes for internal evaluation. Do treat the ability to run rigorous internal evals as a strategic capability, and invest in it accordingly. The enterprises that will iterate their AI systems into competitive advantage are the ones that build the measurement infrastructure to know whether the iteration is working. The ones that don't will be running foundation models from 2024 in 2028, because they cannot tell whether upgrading is safe.
The evals are the product because the eval is what makes the product improvable, defensible, and portable. Without one, an LLM system is a fragile bet on a specific vendor's specific model at a specific moment. With one, the system becomes a compound asset — the golden dataset grows, the metrics tighten, the team's judgment about what works and what doesn't accumulates in a form that outlasts any specific engineer's tenure. That is the difference between an AI feature and an AI capability, and it is the reason serious teams are shifting where they spend engineering hours: not on more feature velocity, but on more evaluation infrastructure.
Adam Valine leads Veil Consulting's Agent & LLM Systems Design practice. Previously twenty years across product and engineering leadership at SaaS and platform companies; currently AI Strategy Lead at Solutions Plus Consulting.
References
- Stanford CRFM: HELM (Holistic Evaluation of Language Models) — Multi-year benchmark suite comparing models across scenarios and metrics. The most widely-cited academic reference for LLM evaluation methodology.
- Anthropic evals repository (GitHub) — Public evaluation methodologies for specific model-behavior conditions.
- OpenAI evals framework (open source, GitHub, 2023) — Template for a serious evaluation pipeline; downstream of most enterprise eval practices.
- LMSYS Chatbot Arena — Millions of human-preference comparisons; the largest open dataset of paired model evaluations.
- USENIX NSDI '23: Agrawal et al. — foundational systems paper on LLM evaluation — Representative of the growing academic literature on evaluating LLM systems in production.
- Anthropic: Prompt Caching documentation — Cited as an example of documentation depth relative to eval tooling on the same provider surface.
- Anthropic: Tool Use documentation — Same context; provider tool-use surface as a comparison point for eval-tool investment.