How to Evaluate an LLM for Enterprise Use

Every serious model now clears the public exams, which is exactly why the public exams have stopped telling you anything. The question worth answering is how a model behaves on your material, at the edges, where the answer is not clean.

A selection committee sits in a conference room with a printout of a leaderboard on the table, and the conversation has the strange quality of an argument nobody can win. The top handful of entries are separated by a couple of points on a composite score, the ordering has shifted twice since the last meeting, and someone has helpfully noted that a different leaderboard puts a different model on top. Everyone in the room understands that the decision matters — this model will sit behind the contract triage, the claims summaries, the internal knowledge assistant, the thing the field team will use forty times a day — and everyone also understands, somewhere below the level of what gets said out loud, that the document on the table is not actually capable of deciding it. So the room falls back on the things it can defend in writing: the score, the price, the name of the vendor. The meeting ends. Nobody has learned anything about how the model will behave on a single one of the company's own documents.

This scene repeats itself in enterprises constantly, and it is worth being precise about why it fails, because the failure is not that the people in the room are lazy or credulous. It is that they are using an instrument built for a purpose that has nothing to do with theirs. Public benchmarks exist to advance a research field. They are shared, they are visible, and they are the thing every lab in the world is explicitly working to improve. That makes them excellent as a coordination device for research and nearly useless as a procurement instrument, for the same reason a standardised test stops sorting applicants once every applicant has spent a year studying for it. The measurement has been absorbed into the thing it was measuring.

The exam everyone studies for stops sorting the candidates

The mechanism here is not scandal, and it does not require anyone to cheat. When a metric is public and reputational, effort flows toward it — training data gets curated with those tasks in mind, evaluation harnesses get tuned, the failure modes that show up on the exam get fixed first because those are the failures that are visible. That is rational behaviour by everyone involved, and its predictable consequence is compression at the top. The frontier options converge on the benchmark not because they have become identical but because the benchmark has stopped discriminating between them. What remains at the top of the table is a few points of spread that fall well inside the noise of your actual workload, and beneath that thin margin sit real differences in behaviour that the exam was never designed to surface.

Meanwhile the things that will actually determine whether the deployment succeeds are, almost by construction, absent from any public set. A benchmark is built from material that can be shared, which means it cannot contain your contract language, your claims notes, your internal shorthand, the three incompatible ways your systems spell a customer's name, or the particular species of document your operation produces by the thousand. It is scored against a clean key, which means it rewards the confident answer and has very little to say about what a model does when the correct response is a refusal, a hedge, or a question back. And it is a snapshot, which means it says nothing about the drift you will experience when a version changes underneath you six months into production. You are choosing an instrument by reading a specification written for someone else's job.

The gap between those two things — the public score and the operational reality — is a good part of why so many ambitious programmes stall after the pilot. Gartner has predicted that over forty percent of agentic AI projects will be canceled by the end of 2027, pointing among other causes to unclear business value and inadequate risk controls. Both of those are, at root, evaluation failures. A programme with unclear value is usually one that never established what "working" meant on its own material before it started; a programme with inadequate risk controls is usually one that never systematically looked for the behaviours it could not tolerate. The demo passed, the score was respectable, and nobody had built the thing that would have caught the problem early enough to matter.

What you actually need to know lives at the edges

Consider what the difficult moments in a real deployment look like, because they are not the ones a leaderboard rehearses. A request arrives that is genuinely ambiguous — the customer's message could be read as a cancellation or as a complaint, and the two readings lead to entirely different downstream actions. A document lands with a contradiction inside it, where the summary paragraph says one thing and the schedule three pages later says another, which is not an exotic occurrence but the ordinary condition of documents that have been amended twice by different people. A user asks for something that should be declined, or asks in a way that is one plausible-sounding step away from something that should be declined. A query touches material the model genuinely does not have, and the only correct behaviour is to say so rather than to produce a fluent paragraph that reads exactly like the ones that are true.

These are the cases where models that look identical on a public exam diverge sharply, and the divergence is not a matter of intelligence but of disposition. One model resolves the ambiguity silently, picking a reading and proceeding with complete confidence, which is catastrophic in a workflow where the wrong branch triggers a refund or a notice. Another surfaces the ambiguity and asks, which is a small friction forty times a day and a saved incident once a quarter. One model reconciles the contradictory document by quietly preferring whichever statement came last; another flags it, which is the behaviour you want when the document is a lease or a purchase order. One declines the borderline request cleanly, another declines it in a way that makes the legitimate version of the same request impossible, which is its own kind of failure and one that no safety score will show you. None of this is captured by a number. All of it is observable in an afternoon if you have the right hundred examples in front of you.

That is the real object of an enterprise evaluation, and it explains why it cannot be bought. It is a body of material drawn from your own operation — the ambiguous cases your staff argue about, the documents that have burned you before, the requests that should be refused, the queries where the honest answer is "I don't know" — paired with what a competent, careful person in your organisation would actually do with each one. Assembling it means going to the people who handle the work, sitting with the exceptions rather than the happy path, and writing down judgements that have never been written down because they lived in someone's experience. It is unglamorous, it takes weeks, and it is the single highest-leverage artefact most AI programmes never build.

The evaluation set is the asset; the model is the consumable

Here is the reframing that changes how the conference room should have spent its afternoon. The model is the perishable part of the decision. It will be superseded, repriced, deprecated, silently updated, or simply beaten by something that did not exist when the committee met, and the half-life of the choice is measured in months. The evaluation set is the durable part. It survives every one of those events, and each time the landscape shifts it converts an anxious re-litigation into an afternoon's work: run the new candidate against the same material, compare it to what you already know about the incumbent's behaviour on your ambiguous cases and your contradictory documents, and decide with evidence about your own operation rather than someone else's exam.

That asset compounds in ways a score never does. It becomes the regression suite that tells you when a version change has quietly altered a behaviour you depended on. It becomes the routing evidence behind a well-run LLM Gateway, where different classes of work go to different models because you have measured which one holds up on which of your material rather than assuming a single winner across everything. It becomes the specification a Reasoning Core is held to when an autonomous system starts acting rather than merely answering, and it becomes the map of where Human-in-the-Loop review genuinely earns its cost — which is precisely at the edges the evaluation set has already identified. The discipline of building it is a large part of what separates organisations that are experimenting from the ones that are actually operating, which is the distinction the category publication covering the autonomous enterprise keeps returning to in its reporting on programmes that made it past the pilot.

So the honest reason most organisations reach for the leaderboard is not that they believe it. It is that the alternative is real work — the slow, specific, internally political work of writing down what good judgement looks like in your business, on the cases where reasonable people disagree. Companies that do it stop asking which model is best and start asking which model is best on this, for us, right now, and find that the question has an answer they can actually defend. The leaderboard tells you what a model can do in a room full of strangers. Your evaluation set tells you what it will do in yours, and that is the only version of the question your business was ever really asking.