Why chat-with-your-data fails in production

What breaks when the demo meets real data?

A demo runs on a clean extract that someone curated the night before. Production runs on the real thing: half-migrated CRM fields, an ERP with fifteen years of custom columns, spreadsheets that are the true system of record for pricing, and three definitions of revenue depending on who you ask. A chat interface does not resolve any of that ambiguity. It samples from it.

The first symptom is two people asking the same question and getting different numbers. The second is an answer that is confidently wrong in a way nobody can trace. After that, trust is gone, and a tool nobody trusts is a tool nobody uses.

Who is allowed to see what?

The demo account can see everything. Your company cannot work that way. Sales should not read payroll. A regional manager should not query another region's margins. The moment a chat tool sits on top of pooled data, it becomes a permissions bypass: the model happily summarizes rows the person asking could never have opened in the source system.

Production-grade systems solve this by inheriting permissions from the systems of record. Retrieval is scoped to what the requesting user may see, agents act through credentials scoped to the task, and the same question from two people can legitimately return two different answers because their access differs.

Where is the evidence?

An answer without sources is an opinion. In operations, opinions do not survive the first dispute with a customer, an auditor, or your own CFO. The demo never shows what happens when someone asks: which rows produced this number, as of when, under which filter?

• Every answer should cite the records and documents it used
• Every metric should trace to a governed definition, not to whatever the model improvised
• Every action an agent takes should be logged with the evidence the approver saw
• Unanswerable questions should be refused and logged, not guessed

Who is accountable when it acts?

Reading data is the easy half. The value is in moving work: correcting an order, drafting the customer email, opening the ticket. The demo either avoids actions entirely or fires them with no gate. Neither survives contact with a real business, where a wrong write to the ERP costs more than the tool ever saved.

The pattern that works is drafts with a named approver. The system detects, explains, assembles evidence, and prepares the action. A person approves it, and the approval is recorded with what they saw. That single design choice converts an unaccountable toy into an auditable coworker.

What does production-grade actually look like?

1. A governed data layer: agreed metric definitions, scoped retrieval, permissions mirrored from source systems
2. An evidence layer: citations, lineage, prompt and tool-call logs, evaluation results tracked over time
3. An action layer: drafted actions, human approval gates, scoped credentials, an auditable history
4. An operations layer: scheduled evaluations, cost and quality monitoring, a way to roll models and prompts forward safely

None of this requires exotic technology. It requires treating the AI system like any other production system: designed for the people who must trust it, not for the person giving the demo.