Business intelligence is moving from dashboards to agents. Instead of opening a report and applying filters, a business user asks a question in plain language and an agent answers it. The interface is better. The architecture question underneath it is harder: how should an AI agent query enterprise data without returning confident, wrong numbers?
| Approach | Compile-time governance | Multi-warehouse | MCP-native | Metric drift protection | Join-path proof |
|---|---|---|---|---|---|
| Raw text-to-SQL (naked LLM) | No | Any | No | No | No |
| Snowflake Cortex Analyst + Cortex MCP | Yes, within Snowflake | Snowflake only | Yes | Yes, within Snowflake | Partial |
| Databricks Genie + Unity Catalog | Yes, within Databricks | Databricks only | Yes | Yes, within Databricks | Partial |
| dbt Semantic Layer + dbt MCP | Yes, metric-level | Warehouse-agnostic | Yes | Yes | Requires modeling |
| Cube + MCP | Yes | Warehouse-agnostic | Yes | Yes | Requires modeling |
| Colrows (semantic execution layer) | Yes, RBAC/ABAC/RLS/CLS at compile time | Yes, 16+ engines | Yes, plus REST | Yes, deterministic | Yes, proven at compile time |
Warehouse-native layers are excellent inside a single platform. Multi-warehouse enterprises need a warehouse-agnostic layer that composes with any of them. Before the comparison matters, though, you have to understand why the naked-LLM row fails, because that is the row most first BI pilots actually ship.
Why "connect an LLM to a database" fails in production
The easiest demo in analytics is to hand an agent a database connection and let it generate SQL. On a small schema with curated questions it looks like magic. Then the schema gets real, the questions get real, and three structural failures show up, none of which are the model's fault.
Domain-specific metrics the model cannot interpret. "Net revenue" means one thing in Finance, another in Sales, another in Product. Databricks put it well, as quoted by Definite: "You ask three executives at the same company to define profit. You will get three different answers." A model reading column names has no way to know which definition is the approved one, the same trap ERP teams document in this analysis of text-to-SQL over ERP data.
Non-determinism. The same question produces different SQL, and different numbers, on different runs. This is metric drift, and it is the failure that ends up in front of the CFO. As the team at Omni Analytics describe it: "The SQL runs without errors and the number looks reasonable. It's wrong."
Wrong joins on multi-table queries. The model joins orders directly to products instead of routing through the subscription model, and returns total order value where the question needed monthly recurring revenue. It is a plausible join. It is the wrong join. This is the same reliability gap examined in deterministic vs. probabilistic text-to-SQL.
The benchmark evidence: the enterprise accuracy cliff
The gap between the demo and production is measurable, and large. On the Spider 2.0 benchmark, built from enterprise schemas that average around 800 columns per database across multiple SQL dialects, models score roughly 6 to 25 percent, near 21.3 percent under multi-step agentic evaluation. The same class of models scores over 90 percent on the original, toy-sized Spider 1.0, a contrast catalogued across the Spider 2.0 literature. That collapse from more than 90 percent to the low 20s is the entire story of why the demo lies. We unpack it in depth in the text-to-SQL accuracy cliff.
The BIRD benchmark lands around 73 percent execution accuracy, but BIRD is less enterprise-shaped than Spider 2.0, so it flatters real-world expectations. Live enterprise SQL is worse still: LiveSQLBench-Large puts models at 30 to 36 percent, and GPT-5 on BIRD-Interact scores 29 percent agentic and 14.5 percent conversational.
State your evaluation regime whenever you quote any of these, because the benchmarks themselves are noisier than the leaderboards suggest. A 2026 UIUC study on annotation errors found a 52.8 percent annotation error rate in BIRD Mini-Dev and 62.8 percent in Spider 2.0-Snow. The honest reading is not "AI cannot write SQL." It is that raw text-to-SQL accuracy on real enterprise data is low and hard to even measure, so you should not stake a boardroom number on it.
The fix is not a bigger model. It is a semantic layer.
Here is the result that should reset the budget conversation. dbt Labs published a 2026 benchmark comparing raw text-to-SQL to semantic-layer-grounded querying. On a well-modeled project, Claude Sonnet 4.6 moved from 90.0 percent raw to 98.2 percent grounded, and GPT-5.3-Codex moved from 84.1 percent to 100.0 percent. On messier real-world data, dbt's internal testing puts raw-schema text-to-SQL near 40 percent against 83 percent grounded in the semantic layer. The academic companion paper, Ganz and Perigaud's 2026 study of semantic layers for reliable LLM analytics, reaches the same conclusion from a different direction. Vendor benchmarking echoes it too: one 2026 study across commercial models reported hallucination rates between 15 and 52 percent on structured analysis tasks, though vendor figures should always sit behind the dbt and arXiv sources.
The same models. A different context. dbt's own summary is the sentence to remember: "When the semantic layer can't answer a question, it says so. Text-to-SQL gives you a wrong answer that looks right." A failure you can see beats a wrong number you cannot. For the mechanism behind that guarantee, see what a semantic compiler actually is.
Fix the Context, Not the Model. It is not that the model is wrong. It is that the model was given no way to be right. A semantic layer gives it that way: one definition, one compilation path, one answer, every time.
How the failure actually unfolds
The reason this pattern keeps repeating is that it fails slowly, in a sequence that looks like success right up until it does not.
The demo works. A small schema, a handful of curated questions, one warehouse. The agent produces plausible SQL and a believable chart. Everyone in the room claps, and a project gets funded.
The pilot works less well. Now the schema is real and the questions are the ones people actually ask. The join is sometimes wrong. The numbers do not quite tie out to the existing dashboard, and someone starts keeping a spreadsheet of discrepancies.
Production breaks. Finance sees revenue in one report and bookings in another, both labeled "revenue." The CFO asks why. The team cannot explain why the same question produced two different SQL queries yesterday and today, because nothing recorded the reasoning and nothing constrained it.
The team blames the model. This is the expensive mistake. They swap in a bigger model, or fine-tune, or add more prompt instructions. Accuracy barely moves, because the model was never the bottleneck. The context was.
The fix is a semantic layer. One definition of each metric. One compilation path. One answer, every time, regardless of which agent asked or when. The dbt benchmark is what that fix looks like in numbers; the boardroom calm afterward is what it looks like in practice.
Where MCP fits, and where it stops
The Model Context Protocol is the reason this conversation is urgent now. It gives agents a standard way to reach analytics tools, semantic services, governed data, and reporting actions, the same shift covered in enterprise MCP adoption. BI is no longer only human-facing dashboards; the same capabilities can be exposed to agents, copilots, and automated workflows through one protocol.
But MCP standardizes access, not meaning. This is the crux, and it is where most BI pilots go wrong: they treat the protocol as the whole answer. Standardizing how an agent connects to the warehouse is the easy half. Standardizing what the numbers mean, so every agent computes revenue the same way, is the harder problem the protocol does not solve. An agent can connect to a warehouse through MCP and still choose the wrong join, still apply the wrong definition of revenue, still read data the user should never see. That gap is the whole subject of why MCP is not enough without a governed semantic layer.
How a BI agent should actually query enterprise data
The safe pattern separates connectivity from meaning. The agent does not invent SQL from scratch. It expresses intent, "show revenue by region for Q2," or "explain why churn increased in enterprise accounts," and a governed layer turns that intent into a trusted answer through a fixed sequence:
- Understand the question: identify the metric, dimension, time period, and audience behind the request.
- Check identity: determine who is asking, and what role, region, or data domain applies to them.
- Route through MCP: reach approved analytics tools and semantic services over the standard protocol.
- Resolve semantics: map the question to approved definitions, valid joins, and metric logic, not column-name guesses.
- Enforce policy before execution: apply RBAC, row-level filters, and column masking before any SQL runs.
- Compile governed SQL: emit a deterministic, dialect-correct execution plan.
- Return trusted insight: deliver the answer with lineage, caveats, and explainability attached.
That moves BI from "the AI wrote a query" to "the AI followed a governed analytical process," which is the difference between a demo and a system a data leader will defend to an auditor.
Access control has to happen before the query runs
Dashboard-era BI enforces access at the report, dataset, or warehouse level. Agents break that model, because an agent asks flexible questions, combines sources, and can summarize sensitive information on request. The same question needs different answers for different people: an executive sees company-wide metrics, a regional manager sees only their region with customer details masked, a support analyst sees ticket metrics but not contract values, and an autonomous workflow gets only the minimum data required to act. The agent must not decide those rules. A governance layer must, evaluated at compile time, which is precisely the discipline behind governing AI agents that query enterprise data and a semantic control plane.
BI outputs will expand well beyond dashboards
Getting the query layer right changes something the dashboard era could not: the output stops being a static chart. Once an agent can ask a governed question, the answer can take whatever shape the moment needs, a dashboard generated on demand, a natural-language explanation of why a number moved, an alert that arrives with causal context instead of just a threshold breach, an executive summary assembled from governed metrics, a workflow trigger fired only when a policy-approved condition is met, or an answer embedded directly inside a CRM or an operational tool. BI becomes active. It moves from reporting what happened to helping a team decide what to do next. But every one of those outputs inherits the trust of the layer underneath it, which is exactly why the layer, not the output format, is where the engineering effort belongs.
What enterprises should evaluate before shipping
These are the questions that separate agentic BI from a risky text-to-SQL demo. Ask them of any vendor, including us:
- Are agents querying raw tables, or governed semantic services?
- Are metrics defined once and reused across every agent and dashboard, or re-invented per query?
- Can access rules be enforced before a query runs, not after the data is already assembled?
- Can generated SQL be reviewed, logged, and audited after the fact?
- Can the system explain which definitions and sources produced a given answer?
- Can an ambiguous question be clarified instead of silently guessed?
- Does the semantic definition travel across warehouses, or is it locked to one vendor's dialect?
If the honest answer to most of those is no, the deployment is a demo wearing a production badge. This is the same evaluation discipline behind picking any agent data stack, and it is why how the semantic layer is exposed over MCP matters as much as which model sits in front of it.
The warehouse-native options, honestly
Several vendors already put a governed semantic layer behind MCP, and inside their own platform they are strong. Snowflake's Cortex MCP server serves Cortex Analyst, Cortex Search, and Cortex Agents with RBAC per tool and built-in OAuth, though it caps at 50 tools with recursion depth 10 and governs Snowflake data only. Databricks Genie exposes agentic analytics with inline visualizations under Unity Catalog governance, within Databricks. The dbt Semantic Layer MCP server exposes metrics, lineage, and trust signals to any agent, and it is warehouse-agnostic if you invest in the modeling. Cube, used by more than 400 companies, offers SQL, REST, GraphQL, and MCP interfaces with row-level, multi-tenant access control; Brex chose it for an embedded AI financial analyst after evaluating dbt Semantic Layer and LookML, a decision the LookML versus dbt Semantic Layer comparison gets into. Dremio enforces fine-grained access control in the query engine on every MCP query.
A good sign the market is converging: the Open Semantic Interchange, launched in 2025 with dbt Labs, Snowflake, and Salesforce, defines vendor-neutral YAML semantic definitions built on MetricFlow. Portable semantic definitions are worth requiring when you evaluate any vendor, so you are not locked to one warehouse's dialect of "meaning."
Where Colrows fits
Colrows is a deterministic semantic execution layer: the same business question compiles to the same governed SQL every time, so metric drift does not happen by construction. It proves join paths at compile time, so the wrong-join, wrong-number failure is caught before execution rather than discovered in a board deck. It enforces RBAC, ABAC, and row and column policy at compile time, and it runs across 16-plus SQL engines, Snowflake, BigQuery, Redshift, Databricks, Postgres, MySQL, ClickHouse, Trino, and more, so a multi-warehouse enterprise gets one governance model rather than one per platform. It exposes MCP tools so agents query BI safely over the protocol enterprises are already adopting, and a REST surface over the same governed graph for everything that is not an agent. The distinction is not that other layers are wrong; warehouse-native layers are excellent in place. It is that compile-time governance, deterministic SQL, and warehouse-agnostic execution are the specific properties a multi-warehouse BI estate needs, and the pattern only works with both MCP and a semantic layer behind it. See the Colrows MCP integration guide for the tool catalog and OAuth flow.
Frequently asked questions
Can I just connect ChatGPT to my data warehouse?
You can, and the demo will look impressive on a small schema. In production, a naked LLM over raw tables picks wrong joins, guesses metric definitions, and returns different SQL for the same question on different runs. It needs a governed semantic layer between the model and the warehouse, not just a database connection.
Why do two teams get different revenue numbers from the same AI dashboard?
Because each agent invents its own SQL at runtime. Without one governed definition of revenue, the same question compiles to different joins and filters, so Finance and Sales see different numbers. A semantic layer defines revenue once and every agent reuses that definition.
What is metric drift?
Metric drift is when the same natural-language question produces different SQL and different numbers across runs, users, or agents, because the metric was never defined in one governed place. It is the core reliability failure of raw text-to-SQL in production.
Is text-to-SQL accurate enough for the boardroom?
Not on its own. On enterprise-shaped schemas, Spider 2.0 shows models scoring roughly 6 to 25 percent under agentic evaluation, versus over 90 percent on the toy Spider 1.0. Grounding the same models in a semantic layer moves dbt's 2026 benchmark from 90.0 to 98.2 percent for Claude Sonnet 4.6 and from 84.1 to 100.0 percent for GPT-5.3-Codex.
What is the difference between text-to-SQL and a semantic layer?
Text-to-SQL asks a model to write a query directly against raw tables. A semantic layer maps the question to pre-approved metrics, dimensions, and join paths, then compiles governed SQL. Text-to-SQL gives you a wrong answer that looks right; the semantic layer either answers correctly or says it cannot, which is the safer failure.
Does Snowflake Cortex Analyst solve the problem?
Within Snowflake, yes: Cortex Analyst plus the Cortex MCP server gives governed, semantic-view-grounded querying with RBAC per tool. It is warehouse-native, so it governs data inside Snowflake only. Multi-warehouse enterprises need a warehouse-agnostic layer that composes across Snowflake, BigQuery, Databricks, and the rest.
What accuracy should I expect from an AI BI system in production?
It depends entirely on the setup, and you should always state the evaluation regime. Raw text-to-SQL on real enterprise schemas can sit near 40 percent; the same questions grounded in a well-modeled semantic layer reach the 80s to high 90s in dbt's testing. Note that the benchmarks themselves carry high annotation error rates, so treat any single number with caution.
How does MCP relate to business intelligence?
MCP is the transport that lets AI agents reach analytics tools, semantic services, and governed data in a standard way. It standardizes access. It does not standardize meaning. Trusted BI needs MCP for connectivity plus a governed semantic layer for correctness.
