# The Best AI Model for Data Analysis Is the Wrong Question

*Guide — 2026-09-05 — by Mahmoud Zalt*

Bare models score 6-16% on the hard split. The same models inside an agent framework reach 45-90%. The scaffold is worth more than the model.

**TL;DR.** On the most realistic public data analysis benchmark, bare models score between 6% and 16% on the hard split. Agent frameworks running over those same models reach between 45% and 90%. The scaffold, meaning the tools, the retries, the memory and the loop, is worth several times more than which model you picked. That is the most defensible finding in this entire area, and it argues for buying a system rather than choosing a model.

## What the data analysis benchmark actually asks

The benchmark worth knowing here is DABStep, built jointly by Adyen and Hugging Face. It contains more than 450 tasks derived from a real financial analytics platform, and it is deliberately not a quiz. Each task requires code-based processing over structured data, CSV and JSON files, combined with contextual reasoning over unstructured documentation that explains what the columns mean.

That combination is what makes it realistic. Real analysis is rarely blocked by the arithmetic. It is blocked by knowing that this fee field excludes refunds, that this flag was renamed last year, and that the definition you need lives in a paragraph of prose rather than a schema. The benchmark splits its tasks into easy and hard, and hard is 84% of the set.

One more number sets the scale. Human analysts score around 62% on the easy split, and that is after three or more hours of work. This is not a benchmark where humans breeze through and models struggle. It is genuinely difficult work for everyone.

## The gap that should change how you buy

Here is the finding. Models evaluated on their own, with no agent framework around them, score between 6% and 16% on the hard split. One reasoning model reached 76.39% on the easy split and only 14.55% on the hard one. Put the same class of models inside a scaffolded agent system and hard-split performance runs from 45% all the way to 90%.

| Who is answering | Easy split | Hard split |
|---|---|---|
| Bare models, no framework | Not reported as a range | 6% to 16% |
| One bare reasoning model | 76.39% | 14.55% |
| Agent frameworks over the same models | Not reported as a range | 45% to 90% |
| A human analyst | ~62% after 3+ hours | Not reported |

Read those two rows next to each other again. The difference between a model answering alone and the same model answering inside an agent framework is larger than the difference between any two models on the list. By a wide margin. Model choice is simply not the dominant variable in data analysis.

## At a Glance

- **6-16%** Hard-split score for models evaluated on their own, with no agent framework
- **45-90%** Hard-split score for agent frameworks running over those same models
- **84%** Share of the benchmark's 450+ tasks that sit in the hard split

It is worth sitting with how unusual that is. In most software decisions, the expensive component is the one that determines the result. Here the expensive component, the frontier model, accounts for a minority of the variance, and the cheap structural decisions around it account for most of it. Teams that spend three weeks comparing models and one afternoon on tooling have the ratio exactly backwards.

## Why a scaffold beats a smarter model

Because analysis is a loop, not an answer. A bare model gets one pass at a hard question: read the prompt, think, produce a number. If it misreads a column definition on that pass, everything downstream is confidently wrong and nothing catches it.

An agent framework does what an analyst does. It opens the documentation, writes code, runs it, sees the error, reads the actual column names, fixes the query, sanity-checks a total against a known figure, and only then answers. None of that requires a smarter model. It requires the model to be allowed to look, run, fail and retry.

This also explains the strange shape of the easy-versus-hard results. Easy tasks are answerable in one pass, which is why a bare reasoning model can reach 76.39% on them. Hard tasks need several passes with real feedback in between, which is why the same model collapses to 14.55%. Buying a better model raises your one-pass ceiling. Buying a loop changes which kind of question you can attempt at all.

## What a scaffold is, concretely

The word scaffold sounds abstract until you list its parts. Each of these is a specific capability that turns a single answer into an investigation, and each one is a place where a system either helps you or leaves you exposed.

## Benefits

### Tool access, granted deliberately

The ability to read files, run code and query a source. Enabled per employee, so an analyst can read your warehouse without being able to touch anything else.

### Retries that recover

A failed query should produce a corrected query, not a confident guess. Most of the gap between 14% and 60% is here.

### Memory across runs

Last month's definition of active customer should still be last month's definition. Context that carries forward stops every report starting from zero.

### Approval gates

Reading data is safe. Writing back to a system of record is not. Consequential actions should wait for a human release.

### An auditable record

Every action logged with a screenshot, so a number in a report can be traced back to the query that produced it.

### Plain-English tool rules

Constraints attached to a specific tool that bind every run, such as which tables are off limits or which figures always need a second source.

Notice how few of those are model properties. Tool permissions, retry behaviour, memory, gates and audit trails are all decisions made by the system around the model, and they are the decisions that moved the benchmark by dozens of points. Every item on that list is a thing we had to build rather than buy at [Sistava](/features), which is a fair warning if you are considering assembling it yourself: the model is the part you can get in an afternoon, and the rest is the project.

## What the benchmark's own authors say about it

The most trustworthy thing about this research is how openly its authors describe its limits. They are worth restating, because anyone quoting the 45-to-90 figure at you without them is selling something.

> The data is synthetically generated, so the business context does not reflect actual performance. The scope is narrow, mostly fraud and payment fees in financial services. The baselines are non-optimised and represent a lower bound. Use this alongside other benchmarks, not instead of them.
>
> The benchmark's authors, summarising their own stated limitations

Take the third one seriously in particular. Non-optimised baselines representing a lower bound means the bare-model numbers are the floor, not the ceiling. A well-prompted model with good tools would do better than 6% to 16%. Which is precisely the point of the article: the improvement comes from what you build around it.

The narrow-scope caveat matters too. If your analysis is marketing attribution or inventory forecasting rather than payment fees, treat the exact percentages as a shape rather than a prediction. The relationship between bare model and scaffolded system is the finding that travels. The specific numbers do not.

In practice that means an analyst employee that can read your files and run code, remembers how you defined a metric three months ago, stops and asks before it writes anything back, and leaves a trail you can check when a number looks off. Those are the parts the benchmark showed were worth 30 to 70 points. The model underneath is the part you should be able to change without rebuilding anything. If you would rather hire that shape than assemble it, an analyst is one of the roles you can [hire on Sistava](/hire-ai-employees) with the tools, gates and record already wired in.

## FAQ

### Which AI model is best for data analysis?

The evidence says this is the wrong variable to optimise. On the hard split of the main data analysis benchmark, bare models cluster between 6% and 16%, while agent frameworks over those same models reach 45% to 90%. The spread caused by the framework is several times larger than the spread between models. Pick any capable model, then invest everything else in the system around it.

### What is DABStep and what does it measure?

DABStep is a data analysis benchmark built by Adyen and Hugging Face with more than 450 tasks derived from a real financial analytics platform. Each task requires code-based processing over structured CSV and JSON data combined with contextual reasoning over unstructured documentation. Hard tasks make up 84% of the set. Human analysts score around 62% on the easy split after three or more hours of work.

### Why do AI models score so badly on hard analysis tasks?

Because hard tasks need several passes and bare models only get one. A model answering alone must read, reason and produce a figure without ever running the query or checking a column definition. One reasoning model scored 76.39% on easy tasks and 14.55% on hard ones for exactly this reason. Give it tools, execution and retries and the number moves dramatically.

### What does agent scaffolding actually mean?

It is the machinery around the model: permission to read files and run code, a loop that reacts to errors and retries, memory that carries definitions between runs, approval gates before anything writes back, and a logged record of every step. On the benchmark, this machinery moved hard-split performance from single digits into the 45% to 90% range using the same underlying models.

### Can I trust these benchmark numbers for my own data?

Only as a shape. The authors state plainly that the data is synthetically generated so the business context does not reflect actual performance, that the scope is narrow and mostly covers fraud and payment fees, and that the baselines are non-optimised lower bounds. They recommend using it alongside other benchmarks. The reliable takeaway is the relationship between bare model and scaffolded system, not the exact percentages.

### Should I switch models to get better analysis results?

Almost never first. Related agent research shows harness technique alone is worth 3 to 6 percentage points, wider than most adjacent leaderboard positions, meaning two models within about five points are effectively tied. Add tool access, execution, retries and memory before you touch the model. Then keep the model swappable so the next release is a configuration change rather than a rebuild.

The whole debate about which model is best for analysis rests on an assumption the evidence does not support: that the model is the thing doing the work. On hard tasks it is not. The loop is doing the work, and the model is one component inside it, contributing far less variance than the tools, the retries and the memory that surround it.

Which leads somewhere specific. If the scaffold is worth more than the model, and consistency across repeated runs matters more than a peak score, and the production layers that benchmarks ignore are where things actually break, then what you want is not a model subscription. It is a system that handles tool use, memory, approval gates and retries, keeps an auditable record of every step, and treats the model as a part you can replace whenever a better one ships. That is the bet [Sistava](/pricing) is priced around, per employee doing work rather than per model, and it is the same test to apply to anything else you evaluate.

**Tags:** ai-data-analysis, ai-models, benchmarks, ai-agents, ai-employees