The hard part of agentic analytics isn't the agent

---

LLMs are getting very good at turning questions into queries. That removes friction, but it doesn't make the answers trustworthy. The difficult part is making sure the agent operates with the same definitions, boundaries and business context that the organization already relies on.

There is a familiar demo in analytical AI. Someone opens a chat, asks a business question in plain English, and a model writes SQL behind the scenes. A few seconds later, it returns a number. But what exactly did the system calculate?

Take a simple question:

How much revenue did we make last month?

Booked or recognized revenue, gross or net, and measured on which date? Refunds and business units have to be settled before the number means anything. A query can be technically correct and still answer the wrong business question.

Text-to-SQL has become the easy part

Text-to-SQL isn't trivial. Complex joins, imperfect schemas and ambiguous requests still make correct query construction hard. But once the intended metric, scope and time frame are settled, writing the SQL is no longer the part that decides whether you can trust the answer. Settling them is.

It's worth separating three kinds of correctness:

1. Syntactic correctness — does the query run? Mostly technical.
2. Data correctness — does it retrieve the data the query intended to retrieve? This requires understanding the data model.
3. Business correctness — does the query represent what the company actually means by the question? This requires understanding the business.

That third level is where many of the dangerous failures happen. If two people ask the same question, they should not get two different interpretations. If the number returned by an agent differs from the number in a trusted business report, the agent has failed, however convincing the interaction looked.

A polished wrong answer is worse than no answer at all.

Your agent needs a semantic system

A database schema tells an agent what exists physically. A semantic system tells it what exists conceptually: metrics, dimensions, entities and the relationships between them. It also tells it about scope, because some data exists physically and is still unsuitable for analytical use, whether because it is incomplete, experimental or ungoverned.

The semantic layer governs what gets calculated: the definitions the agent resolves instead of recreating. But it's only part of what's needed. The agent also needs a written body of business context that governs what the words mean and how the company works: two internal terms that point at the same concept, a channel in one region that isn't the same process as in another, a decision taken last year that still shapes how a number has to be read. And it needs to know how specific kinds of analysis are done here, in other words a set of skills that governs how the data should be worked with. These documents used to be written for us: to speed up onboarding, to document what had been built, or to reach agreement between teams. That is no longer what they are for. The three of them are now the fundamentals that decide whether the data is AI-ready, and they should be written for agents too.

“What was revenue yesterday?” may need a single retrieval. “Why has conversion fallen this month?” needs clarification, segmentation, comparisons and several hypotheses, in roughly that order. Definitions don't tell an agent that, and business context doesn't either. It's method, and if nobody writes it down the agent improvises it differently every time.

None of the three is documentation in the old sense. It counts only if the agent reads it at the moment it answers. Experienced analysts carry most of this in their heads, which worked while the only consumers were people who could ask each other.

Trust comes from bounded freedom

There is an understandable instinct when building agents: give them access to more tables, more tools, more documents, more autonomy. For analytical systems, that instinct is often backwards.

But reducing freedom is only the mechanism. What makes the result trustworthy is where the limits came from: rules that somebody with the authority to set them agreed on, wrote down, and left open to review. An agent bounded by nothing but a developer's guess is constrained, not governed.

Which is why it matters that a governed metric is resolved through the semantic layer rather than merely shown to the model. If the agent asks the layer for revenue and the layer returns it, the agent can't quietly recreate the definition, because it never writes that part of the query. Where a dimension isn't governed, the agent shouldn't invent one. Ambiguity is best met with a question. And when the answer sits outside the governed perimeter, the agent should be able to say so and stop. That is the honest answer to the obvious objection, which is that a bounded agent can't cover everything. It can't. We spend a lot of time trying to make agents smarter, and not enough time reducing the number of ways they can be wrong. Creativity is useful when exploring hypotheses or suggesting follow-up questions; it is much less useful when deciding what the company means by revenue.

Treat analytical agents as products

Once an agent becomes useful, a new problem appears: how do you change it without quietly making it worse? Business definitions evolve, models change, context gets added, new behaviors are introduced.

A practical response is to maintain a growing set of representative business questions with an expected interpretation agreed in advance. Interpretation is the word carrying the weight. Grading the agent against a reference query is close to useless, because many different queries are correct. Grading it against a reference number is worse, because the number moves with the data, and a right number reached through the wrong metric passes in silence. What you assert on is the resolution: which metric it picked, at which grain, over which date, and what it left out.

You also need to observe real usage: what people are actually asking, which concepts are repeatedly misunderstood, where the agent needs clarification, which requests fail because the semantic system is incomplete, and which unexpected use cases are becoming valuable.

Agents are another interface

Companies have consumed analytical logic through dashboards, spreadsheets, notebooks and reports for years. Agents add another interface, and it may become a particularly powerful one, because they reduce the cost of asking follow-up questions and of moving from a number to an analysis.

If the meaning of a metric lives inside a dashboard, an agent has to rediscover it. If it lives in a governed semantic system, both the dashboard and the agent can consume the same definition.

Agentic analytics does not make semantic modeling important. It exposes how important it always was.

Juan Caravaca — Head of Data and technical data leader. Analytics Engineering, semantic models, governed metrics and the platforms underneath them.