The accuracy cliff, at a glance
Best-published-result accuracy on each benchmark, using strong code-agent frameworks or off-the-shelf LLMs as noted. Higher is better.
The benchmark table
| Benchmark | Best result | System / model | What it tests | Source |
|---|---|---|---|---|
| Spider 1.0 | 91.2% | o1-preview code-agent framework | Cross-domain academic schemas, small and clean | Lei et al., Spider 2.0 paper (arXiv:2411.07763) |
| BIRD | 73.0% | o1-preview code-agent framework | Larger, dirtier databases with values and evidence | Reported in Spider 2.0 paper; BIRD, NeurIPS 2023 |
| Spider 2.0 | 21.3% | o1-preview code-agent framework | 632 real enterprise workflows; schemas averaging ~800 columns | Lei et al., ICLR 2025 (arXiv:2411.07763) |
| BEAVER | ~0% (end-to-end) | Off-the-shelf GPT-4o, Llama3-70B-Instruct | Real private enterprise data-warehouse queries | BEAVER enterprise benchmark |
| Sequeda / data.world | 54.2% (16.7% raw) | GPT-4 zero-shot, via knowledge-graph representation | 43 enterprise questions on an OMG Property & Casualty insurance schema | Sequeda, Allemang & Jacob, data.world (arXiv:2311.07509) |
| dbt Labs 2026 | 100% (84.1% raw) | GPT-5.3 Codex, via the dbt Semantic Layer | ACME Insurance, 11 questions x 20 runs, 15 tables; Claude Sonnet 4.6 moved 90.0% to 98.2% | dbt Labs benchmark, 7 April 2026 |
| Vendor internal (context) | 90%+ | Cortex Analyst on a curated semantic model | Snowflake internal 150-question set (not independent) | Snowflake engineering blog |
The bottom row matters for reading the rest: a curated semantic model lifts a Snowflake-internal set to 90%+, while the same class of single-shot LLM sat at 51% on that set. That is the whole thesis in one line: the semantic model does the work, not the raw model.
How to read these numbers
- Academic benchmarks flatter models. Spider 1.0's schemas are small and clean. Real warehouses are not. The 91.2%-to-21.3% drop from Spider 1.0 to Spider 2.0 is the same class of system on realistic data.
- Enterprise schemas are the hard part. Spider 2.0 databases average roughly 800 columns. Ambiguous names, undocumented joins, and governance rules are the real difficulty, not SQL syntax.
- Off-the-shelf LLMs collapse on private data. BEAVER found near-0% end-to-end accuracy for GPT-4o and Llama3-70B on real warehouse data.
- Curated context is the lever. Every result above the raw baseline comes from better grounding, a semantic model, not a bigger LLM.
Fix the Context, Not the Model. The benchmarks say it plainly: accuracy tracks the quality of the semantic layer grounding the query, not the size of the model writing it. That is where enterprise text-to-SQL is won or lost.
What a benchmark must model to reflect real data stores
The gap between 91% and 21% is not a modelling error. It is the difference between a benchmark schema and a production data store. A benchmark that claims enterprise relevance has to reproduce the conditions below, and most do not.
- Schema scale. Spider 1.0 databases carry a handful of tables. Spider 2.0 averages roughly 800 columns per database, and real warehouses routinely exceed that. Column count alone moves accuracy more than model choice.
- Ambiguous and duplicated names. Production stores accumulate
revenue,revenue_net, andrev_finalacross teams. Choosing between them is a governance question, not a syntax one, and no amount of model scale resolves it. - Undocumented join paths. The correct path between two tables is often tribal knowledge. A model that guesses a plausible join returns a confidently wrong number rather than an error, which is the failure mode that matters most in regulated settings.
- Access control at query time. Real stores enforce row- and column-level predicates per user. A benchmark that runs as a superuser measures a query no production user could ever execute.
- Dialect divergence. The same question compiles differently on Snowflake, Databricks, and BigQuery. Single-engine benchmarks hide the portability cost entirely.
- Semantic drift. Definitions change after the benchmark is frozen. Production accuracy decays unless something detects and repairs that drift.
Spider 2.0 and BEAVER score low precisely because they model several of these. That makes them harsher benchmarks and better predictors. Treat any vendor accuracy figure that omits these conditions as a measurement of the benchmark, not of the system.
Closing the gap: the semantic layer
The numbers explain why deterministic, semantic-layer approaches exist. A typed semantic graph resolves meaning, proves the join path, and refuses on ambiguity instead of guessing. Colrows is built on this compile-then-execute model: it emits deterministic, dialect-perfect SQL across 16+ engines with governance enforced before execution. For the narrative version of this data, see the text-to-SQL accuracy cliff; for the tool landscape, see the best text-to-SQL tools; for the architecture, see deterministic vs probabilistic text-to-SQL.
Cite this benchmark
You are welcome to reference this compilation. Suggested citation:
Primary sources: the Spider 2.0 paper (Lei et al., arXiv:2411.07763) reports the Spider 1.0, BIRD, and Spider 2.0 figures for a common o1-preview code-agent framework; the BEAVER benchmark reports the near-0% off-the-shelf result on real warehouse data. Always verify against the primary papers before publishing.
Frequently asked questions
What is the best text-to-SQL benchmark for enterprise use?
Spider 2.0, for its realism: 632 real workflows over schemas averaging ~800 columns. Best published result is about 21.3%. BEAVER tests real warehouse data, where off-the-shelf LLMs score near 0% end-to-end.
Why is enterprise text-to-SQL so much harder than benchmarks suggest?
Academic sets use small, clean schemas; real enterprises have hundreds of columns, ambiguous names, and governance rules. The grounding semantic model, not the LLM, is the bottleneck.
How do you improve accuracy on real data?
Ground generation in a governed semantic layer that resolves meaning, proves joins, and refuses on ambiguity. Fix the context, not the model.
