How to Pick the Best AI Model for Customer Support

What a customer support benchmark actually measures

The benchmark people cite for this question is tau-bench, and its successor tau2-bench, built by the team at Sierra. It does not ask a model to answer a question well. It drops the model into a multi-turn conversation with a simulated customer, hands it real API tools that read and write a live database, and grades whether the customer left satisfied while the agent obeyed a written company policy.

That last clause is the whole test. Three domains are covered: retail, airline and telecom. The second version adds what the authors call dual control, where the customer can also call tools, which is how technical troubleshooting really works. Performance drops noticeably when an agent moves from acting alone to working with a customer who is also pressing buttons.

So when someone tells you a model is good at support, the first question is which domain they measured. The answer moves the number more than the choice of model does. That is not a criticism of the benchmark. It is the benchmark's most useful finding.

The 20-point gap has nothing to do with language

The single most useful fact on the board is that the same model scores roughly 20 points worse on airline scenarios than on retail ones, and it does so consistently down the entire leaderboard. Refund requests and flight changes are written in the same plain English. What differs is the policy behind them.

Airline policy has side effects. Changing a booking touches fare rules, cabin class, baggage entitlement and a refund calculation at once, and getting any one of them wrong invalidates the whole resolution. A retail return is usually a single decision with a single consequence. If your support policy looks more like the airline column, expect airline-shaped results whichever model you pick.

This is the most actionable finding in the whole area, because policy complexity is the one variable you control. You cannot make a model smarter. You can make a rule unambiguous, split a compound decision into two steps, and remove the clause that three humans already interpret differently.

The score that matters is the eighth attempt, not the first

Almost every headline score you have read is a single-attempt number. The same researchers also measured what happens when the identical scenario runs several times over. One GPT-family model scored under 50% on first attempt and below 25% when those same scenarios were run eight times.

That is not a rounding error, it is a different property being measured. Capability is what a model can do on a good attempt. Consistency is what it does every time. Support is a job that runs near-identical scenarios hundreds of times a week, and a queue does not care about your best run.

The same research shows that harness technique, meaning how tools are described and how the loop retries, is worth 3 to 6 percentage points on this benchmark. That is wider than most adjacent positions on the leaderboard. Two models within about five points of each other are tied, and treating them as ranked is reading noise as signal.

None of this makes model choice irrelevant. It makes the choice quickly exhausted. Once you are inside the top group, everything left is engineering: how the policy is written down, which tools the agent may touch, what happens when a call fails, and who reviews the runs that went sideways. That is where the remaining points live, and unlike the model, it is yours to improve.

For support, latency is a product feature

Support is the one job where time-to-first-token belongs on the requirements list rather than the infrastructure notes. A customer staring at an empty chat window for a minute has already decided the answer will be useless. The published leaderboard shows the spread plainly, and it is enormous.

• First reply speed. The quickest models start answering in roughly 0.31 to 0.41 seconds. Top-end reasoning models take 60 seconds or more, and one takes over 170. In a live chat those are different products, not different tiers.
• Blended cost. Roughly $0.40 to $2.34 per million tokens across the same board, with a throughput gap of about six times running in the same direction. The cheap end is also the fast end.
• Policy compliance. Scored by the benchmark, and the place where the domain gap shows up. This is the number worth optimising, because it maps directly onto tickets you would have had to reopen.
• Escalation behaviour. Not measured at all by any public score. Whether an agent hands off cleanly when it is out of its depth is something you have to test yourself, on your own tickets.

A slow reasoning model can be the right answer for a complex billing dispute that arrives by email once a day. It is the wrong answer for a chat widget. Most support operations need both, routed by ticket type, which is another argument for a system that can hold more than one model rather than a single subscription.

That routing is why Sistava lets you choose the model behind each employee instead of committing the whole account to one. A fast one answers the chat widget, a slower and more careful one takes the disputes that arrive by email, and neither decision locks in the other. It also makes the swap you will want in six months a setting rather than a migration.

There is one more reason not to over-index on the board: it is frozen from late 2024, and most current models were never run on it. Independent reproductions exist but use different harnesses, so their numbers are not directly comparable. Treat every figure here as the shape of the problem, not a shopping list.

How to choose without waiting for a leaderboard

A test you can run on your own queue this week

1. Write the policy down before you test anything — Refund windows, exceptions, who may authorise what, and what is never allowed. If your team disagrees on a clause, no model will resolve it. Ambiguity in the policy is the airline column in disguise.
2. Rebuild your ten worst tickets as scenarios — Not your average ticket, your painful ones. The compound requests with three side effects are the ones that separate a usable agent from a demo.
3. Run each scenario eight times, not once — Score the worst run, not the best. This is the single change that makes an internal evaluation predict production behaviour instead of flattering it.
4. Measure time to first token separately — Full completion time is an operations metric. First-token time is what the customer experiences as responsiveness, and it is the one you should hold a threshold against.
5. Price the whole loop, not the model — Retries, tool calls and re-reads of context dominate the bill in agentic support. A cheaper model that needs three attempts is not cheaper.
6. Keep the model swappable — Whatever wins today will be second in six months. If changing models means rewriting your tooling, you have bought a dependency rather than a capability.

That routine takes a couple of days and outperforms any amount of leaderboard reading, because it tests the exact thing you will ship: your policy, your tools, your customers, your definition of resolved.

Pricing the whole loop is a great deal easier when the loop is priced as one thing. Plans at Sistava meter the retries, the tool calls and the context re-reads as credits included in the plan, so a support queue has a monthly number you can forecast rather than a per-token bill that only resolves itself at the end of the month. Whatever you use, count a full week of real tickets before you decide anything is cheap.

The practical version of everything above is a support employee whose tools you enable one at a time, whose limits are written as plain-English rules attached to each tool, and whose every action lands in a reviewable feed. Memory carries across runs, so the agent that handled a customer in March still knows them in September.

That is what we build at Sistava. You hire a support employee, switch on the tools it needs one at a time, and write its limits in plain English rather than in code. The policy work described above is still yours to do, and no product removes it. What changes is that once the policy is written down, something applies it the same way on every ticket, including the one that arrives at three in the morning.

What these numbers deliberately leave out

The benchmark's own authors note a possible bias toward agents built like the vendor's own product, because the vendor sells a support agent. That is an honest disclosure and worth repeating. There is also no vendor-published resolution rate or deflection benchmark anywhere in this space. If a sales deck quotes one, it is a marketing number from a customer sample, not a measured score.

More broadly, public benchmarks grade isolated answers rather than agent behaviour, ignore retrieval context and policy compliance in production, and omit latency, cost, safety and escalation entirely. Contamination in training corpora means scores read better as upper bounds on capability than as clean measurements. They are useful for narrowing a shortlist and insufficient as a release gate.

The honest conclusion from the research is uncomfortable for anyone selling a model ranking. Domain policy moves the score more than model choice. Consistency across repeated runs is worse than the headline and matters more. Harness technique is worth more than several leaderboard positions. Latency, cost and escalation, the three things a support team cares about daily, are not scored at all.

Which points somewhere other than a model subscription. What actually determines your outcome is the system around the model: tool access granted per employee, rules that bind every run, approval gates on anything with a consequence, retries that recover instead of repeating, memory that carries a customer's history forward, and a record you can audit after the fact. Get that right and the model underneath becomes a component you can change on a Tuesday.