This is the gap between a demo and production. Agents are fluent. Fluency is not correctness. The moment you put an agent in a loop, give it tools, and let it act on enterprise data, the small probability of being wrong at each step compounds into a large probability of being wrong overall. This post is about why that happens, exactly where agents fail, and why a deterministic semantic layer is becoming table stakes for any enterprise that wants agents it can trust, audit, and defend.
What an AI agent actually is
Strip away the marketing and an agent is simple: an LLM used as a reasoning engine inside a loop, connected to tools so it can act on the world instead of just describing it.
The canonical pattern is ReAct, "Reasoning and Acting," introduced by Yao et al. in 2022 (arXiv:2210.03629). The agent interleaves three things in a loop:
Thought: I need last quarter's revenue for the EU region.
Act: query_warehouse("SELECT SUM(amount) FROM orders WHERE ...")
Obs: [returns 4,210,553]
Thought: That looks plausible. Now compare to the prior quarter...
Act: query_warehouse(...)
Obs: ...
It keeps looping, Thought then Act then Observation, until it decides the goal is met. Tools can be APIs, database queries, knowledge bases, or a semantic layer. In LangChain this is `create_react_agent` plus `AgentExecutor`. The critical detail, in LangChain's own words: the model "only ever sees the name and docstring" of a tool. It never sees the implementation. It is reasoning about your data and your tools from text descriptions and statistical patterns.
How does the agent reason inside each Thought? Chain-of-thought prompting (Wei et al., arXiv:2201.11903), which generates intermediate reasoning steps and measurably improves multi-step performance. Bigger context windows let it carry more history. Role prompts ("you are a careful financial analyst") shape its behavior. All of it is probabilistic. Every Thought, every Act, every argument is a sample from a distribution.
That single fact, every step is a sample, is the root of every failure that follows.
Where agents fail (and which step goes wrong)
1. Hallucinated tools and arguments
The agent invents a tool that does not exist, or calls a real tool with wrong arguments. This is not theoretical. LangChain practitioners describe agents that confidently call imaginary endpoints. In one documented case an agent hallucinated a `bulk_search_users` endpoint, retried variations (`bulk_user_search`, `search_all_users`), and generated "over 2,000 failed requests" before anyone noticed, plus the cloud bill. Where it fails: the Act step. Why: the model pattern-matched a plausible API name from training data, and nothing told it the tool does not exist.
2. Hallucinated definitions
The agent assumes a metric exists, or assumes its formula. Ask for "revenue" and the agent does not know whether your company means gross, net of refunds, net of discounts, recognized, or booked. It picks one. As Colrows has written elsewhere, an agent over a raw warehouse "does not know the orders table excludes voided transactions; it does not know the finance department uses a different revenue definition than the product team." Where it fails: the Thought step, before any SQL is even written.
3. Silent wrong answers
The worst failure mode, because nothing breaks. The tool runs, returns a plausible number, and the agent reports it. dbt Labs' 2026 benchmark names the mechanism precisely: raw text-to-SQL "continues to fail in subtle ways: misinterpreted column names, broken joins, fiscal calendar confusion, ambiguous metric definitions." Snowflake's framing: a model pointed at raw tables "can write SQL that runs and returns a number that is simply wrong; it guessed the join, or summed the wrong column, or used a metric definition nobody agreed to." No error. No stack trace. Just a wrong number with a confident sentence wrapped around it. This is why a semantic layer built on explicit, governed definitions matters: ambiguity is detected and refused before execution.
4. Probabilistic accumulation, the multi-step killer
Here is the math every buyer and builder needs to internalize. In a loop of n independent steps, each with per-step accuracy a, end-to-end success is a^n. This is not a model flaw. It is multiplication.
| Per-step accuracy | 1 step | 5 steps | 10 steps |
|---|---|---|---|
| 90% | 90% | 59% | 35% |
| 95% | 95% | 77% | 60% |
| 98% | 98% | 90% | 82% |
| 99% | 99% | 95% | 90% |
A 90%-accurate agent, which sounds excellent, completes a 5-step task correctly only 59% of the time and a 10-step task 35% of the time. A recent arXiv study (2601.22290) puts it starkly: even 99% per-step accuracy "degrades to 36.6% at 100 steps." Galileo's analysis: a 5% per-step error rate over 10 steps "degrades end-to-end success to roughly 60%." And the errors do not stay contained, because, as multi-turn research found, "when LLMs take a wrong turn in a conversation, they get lost and do not recover." One bad assumption in step 2 poisons every step after it.
5. Definition drift
Across a long loop, the meaning of "active customer" can shift, between tools, between agents in a multi-agent system, between the start and end of the same task, and the agent never notices. Multiply this across a fleet of agents and you get the fragmentation: one agent defines churn one way, another differently, "consistency erodes silently."
How semantics fixes each failure, mechanically
A semantic layer is not documentation the agent might consult. Done right, it is the compiler the agent's intent must pass through. Here is the failure-to-fix mapping:
| Failure mode | Technical fix from semantics | Business outcome |
|---|---|---|
| Hallucinated tool | Typed, listed, type-checked interface; non-existent tools cannot be called | No runaway API loops, no surprise bills |
| Hallucinated definition | Metric defined once; agent cannot invent a formula | One number, defensible across teams |
| Silent wrong answer | Join paths proven valid at compile time; invalid queries rejected before execution | No silently wrong board numbers |
| Probabilistic accumulation | Each step compiles against the same deterministic model; covered-query accuracy near 100% | a^n collapse avoided on covered scope |
| Definition drift | Versioned definitions; same model every step | Reproducibility across the whole loop |
The accuracy lift is measured, not asserted. dbt Labs reran their semantic-layer-versus-text-to-SQL benchmark in 2026 with current frontier models. The result: "for queries covered by a well-modeled Semantic Layer, accuracy approaches or hits 100%," because "the Semantic Layer's deterministic query generation means the LLM can't produce subtly wrong results." Snowflake describes the same move as replacing "plausible SQL" with "governed SQL," where "the accuracy of the answer moves from the model's guess to a definition a human approved."
Plug those numbers into a^n. An agent compiling through a semantic layer at 98% per-step accuracy completes a 5-step task at 90%. A raw text-to-SQL agent at 85% completes it at 44%. Same model, same task. The difference is whether meaning was guessed or compiled. Multi-agent enterprises need this consistency. When you build a company brain with shared semantic memory, agents stop reinventing definitions and start reasoning over a common ground truth.
And it is cheaper. Fewer hallucinations mean fewer refinement loops. The cost of those loops is real: per the SEA-SQL paper (arXiv:2408.04919, Table 5, BIRD dev set), GPT-4-based iterative text-to-SQL methods cost roughly $0.12 to $0.79 per query (DAIL-SQL $0.12, MAC-SQL $0.22, DIN-SQL $0.79) at 12 to 16 seconds of API time each. Agents typically take 3 to 5 refinement steps, so a single answer can compound into multiple dollars and a minute or more of latency, much of it spent re-trying queries that a deterministic layer would have compiled correctly the first time. SEA-SQL itself shows the alternative direction: by grounding generation in a semantic-enhanced schema, it cut per-query cost to $0.0065 while matching GPT-4-level accuracy.
A concrete comparison
Raw agent, "show me net revenue for EU enterprise customers last quarter":
- Thought: net revenue is probably gross minus refunds. Guesses.
- Act: writes a join from `orders` to `customers` it has never verified.
- Obs: a number. It excludes voided transactions the agent did not know about, and uses the product team's revenue definition, not finance's.
- Reports a confident, wrong figure. No error, no audit trail of why.
Agent over a semantic execution layer, same question:
- Intent compiles through one source of meaning. "Net revenue" resolves to the governed, versioned definition. "EU" and "enterprise" resolve to scoped dimensions.
- The join path is proven valid at compile time, or compilation fails loudly.
- Policy predicates inject automatically: this persona sees only the rows it is authorized to see.
- Dialect-perfect SQL executes. The result carries a reconstructable trace: definition version, join path, policy applied.
One of these you can put in front of an auditor.
Why reliability is now a business and compliance requirement
For the strategic buyer, this stopped being a quality-of-life issue and became a control requirement.
The cost of being wrong is documented. AI hallucinations cost businesses $67.4 billion globally in 2024 (AllAboutAI's 2025 report). Per Deloitte's Global AI Survey, 47% of enterprise AI users made at least one major business decision on hallucinated content. The headlines are no longer hypothetical: Deloitte Australia refunded the final installment of a AU$439,000 (about US$290,000) government contract after its report cited fabricated references and a made-up quote attributed to Federal Court Justice Jennifer Davies, later disclosing it had used an Azure OpenAI GPT-4o tool chain; EY Canada withdrew a report with hallucinated footnotes. IBM's 2025 Cost of a Data Breach Report pegs the global average breach at $4.44 million (US record $10.22 million) and finds shadow AI "added an extra USD 670,000 to the global average breach cost," with 97% of AI-related breaches tied to organizations that "lacked proper access controls."
The regulators now require the trace, not the vibe. This is where deterministic agents stop being a nice-to-have:
- EU AI Act, Article 26(6): deployers of high-risk AI systems "shall keep the logs automatically generated by that high-risk AI system to the extent such logs are under their control, for a period appropriate to the intended purpose of the high-risk AI system, of at least six months." High-risk obligations bind from 2 August 2026 (a provisional 2026 "Omnibus" proposal could extend stand-alone high-risk dates to December 2027, but it is not yet law). An agent over a raw LLM logs tokens; a semantic-grounded agent logs logic, the step, the tool, the definition version, the result.
- SOX 404: if an agent touches financial reporting, controls must cover it. Auditors require "timestamps, approvals, and logs proving controls were actually executed," and the "black box of automation" is already a recognized audit problem.
- HIPAA 164.312(b): clinical systems must "record and examine activity in information systems that contain or use electronic protected health information," with field-level, decision-level logs retained for six years.
- GDPR Article 22 and Articles 13-15: solely-automated decisions with significant effects carry rights to "meaningful information about the logic involved."
Every one of these reduces to a single test: can you replay the decision and prove it was correct under the rules in force at that moment? A probabilistic agent regenerates a different reasoning trace every run; it cannot. A semantic-grounded agent with versioned, point-in-time definitions can.
The competitive angle. Agents without semantic grounding are fine for exploration, prototyping, and low-stakes workflows. The teams pulling ahead are putting semantic-grounded agents into compliance, finance, and production. Deterministic agents are becoming the entry ticket to autonomous work in regulated industries, not a differentiator. To evaluate which approach fits your enterprise, use the semantic layer evaluation checklist to assess vendor offerings against your compliance and autonomy requirements.
The architecture choice: semantics as a tool vs as a foundation
Builders face a real design decision.
Semantics as a tool. The semantic layer is one tool in the agent's kit, exposed over MCP as something like `get_metric("revenue", filters)`. This is a genuine improvement: where a metric exists, the agent gets a governed number. Its limits are structural. Metric stores are "pre-defined metrics only," so the moment an agent needs a novel multi-hop query ("churned EU customers in BFSI whose contract value dropped 20% year over year"), the metric was never authored and the layer cannot resolve it. And the agent can still ignore the tool and hand-write SQL.
Semantics as a foundation. The agent's intent compiles through the semantic layer by default. It is not a sidecar the agent may consult; it is the path the query has to take. Hand-written SQL, hallucinated joins, and stale retrievals are detectable and refusable. This is the model that survives audit, because every query, by construction, carries a trace.
The honest framing: a tool is faster to adopt and good enough for analytics assistants. A foundation is what you need for agents that act, that approve, adjust, and decide.
Where Colrows fits, and where it does not
Colrows is a semantic execution layer built for the foundation model, not the sidecar model. It compiles agent intent through a typed, versioned semantic graph before any SQL touches a warehouse. Every query follows the same path: context resolution, join-path proof, policy enforcement, then governed execution, in that order. Concretely:
- Agent-native interfaces. Any agent that can call HTTP, JDBC, or an MCP tool surface can use it. The agent passes intent; Colrows returns governed, dialect-perfect SQL or executed results across 16+ engines (Snowflake, Databricks, BigQuery, Redshift, Postgres, and more).
- Compile-time governance. Join paths are proven at compile time; ambiguous queries fail compilation rather than returning a plausible wrong answer. RBAC, ABAC, and row/column predicates shape the allowed subgraph per persona.
- Multi-scope meaning. The same string ("revenue") resolves differently across global, datastore, persona, and user scopes, so finance and product can disagree without forking a definition.
- Point-in-time reproducibility. Definitions are versioned; you can re-run a historical query and prove it used the exact rules in force then. That is the EU AI Act, SOX, and HIPAA replay test, satisfied by construction.
- Autonomous maintenance. The graph builds from your warehouses, dbt models, BI tools, catalogs, and documentation, and flags definition drift for human approval rather than breaking silently.
Honest about scope. A semantic layer is deterministic on the scope it covers. Questions outside that scope, genuinely novel exploration, unmodeled data, ad hoc discovery, still need a fallback path, and that path is probabilistic text-to-SQL with the best schema context you can give it. The right production pattern is exactly what dbt recommends: check whether the semantic layer can answer first, and fall back to raw generation only when it cannot. The goal is not to claim semantics answer everything. It is to make the agent deterministic everywhere it matters, and honest about the boundary. For the comparison between different approaches—semantic layer platforms compared—you'll see that all production systems enforce this same principle.
The bottom line
Agents fail in predictable ways: they hallucinate tools, invent definitions, return silently wrong answers, and compound small per-step errors into large end-to-end failures. A semantic layer fixes each of these mechanically, by type-checking the interface, governing the definitions, proving the joins, and versioning the meaning. The technical property, determinism, directly produces the business requirement: auditable, reproducible, compliant agents.
The first wave of enterprise AI asked "what can AI do?" The production wave asks a harder question: "can you prove it?" For autonomous work in a regulated enterprise, the answer has to be yes, and that answer lives in the semantic layer, not the prompt. Build agents that are dialect-perfect and context-aware—agents that understand what your data means and operate within the boundaries your compliance function has set. That is the deterministic agent that survives audit and scale.
Frequently asked questions
What is an AI agent?
An LLM used as a reasoning engine in a loop, connected to tools so it can act; the canonical pattern is ReAct (Yao et al., 2022, arXiv:2210.03629), which interleaves reasoning traces, actions, and observations.
Why do AI agents hallucinate?
Each reasoning and tool step is a probabilistic sample; without a typed interface the agent can invent tool names, arguments, and metric definitions. LangChain agents have been documented calling non-existent endpoints.
What is the multi-step accuracy problem?
For n independent steps each at accuracy a, end-to-end success is a^n; a 90% per-step agent hits 59% over 5 steps and 35% over 10. At 99%, it degrades to 36.6% at 100 steps.
How does a semantic layer make agents more reliable?
It compiles intent deterministically, type-checks tools, governs definitions, and proves joins, lifting covered-query accuracy toward 100% (dbt 2026 benchmark: 98-100% for semantic layer vs 84-90% for raw text-to-SQL).
Semantic layer as a tool vs a foundation?
As a tool it answers pre-defined metrics on request; as a foundation every agent query compiles through it and is refusable if out of scope. A foundation is what you need for agents that act and approve.
Do deterministic agents help with compliance?
Yes. EU AI Act Article 26(6), SOX 404, HIPAA 164.312(b), and GDPR Article 22 all require reconstructable logs and explainable logic that semantic-grounded agents produce by construction.
How much do agent hallucinations cost enterprises?
AI hallucinations cost businesses $67.4 billion globally in 2024 (AllAboutAI). Deloitte found 47% of enterprise users made at least one major business decision on hallucinated content.
Where does Colrows fit?
It is a cross-estate semantic execution layer that compiles agent intent into governed, deterministic, point-in-time-reproducible SQL over MCP, HTTP, and JDBC. Agents interact with it without guessing joins or inventing definitions.
