Your AI pilot didn't fail on the model

A pattern worth naming, because it repeats with unusual consistency.

A company runs an AI pilot. The demo is good — genuinely good, not politely good. Everyone in the room can see the value. Then it goes to production and the answers get subtly worse, support tickets appear, trust erodes over a few weeks, and eighteen months later the project is described as “something we tried”.

The post-mortem usually blames the model. It is almost never the model.

The gap between demo and production

A demo runs on data someone curated. Not deliberately dishonestly — just naturally. Whoever built it reached for documents they knew were good, records they knew were current, examples they knew were representative.

Production runs on everything else. The contract from 2019 that was superseded but never archived. Three versions of the same specification with no indication which is authoritative. Support tickets where the resolution was communicated by phone and never written down. Product records where a field means one thing for items added before a migration and something else after.

The model is identical in both cases. The inputs are not.

An agent reasoning over stale, contradictory or incomplete data is not a smaller version of a good agent. It is a confident source of wrong answers, which is worse than nothing.

Three failures that look like model problems

It answers from documents that were superseded. Nothing in a vector store indicates that one document replaced another. Both match the query. The older one is often written more clearly, because it was written when someone had time. The retrieval layer, given no signal about authority, sometimes prefers it.

It cannot say “I don’t know”. Language models are built to produce plausible continuations. Absent a hard constraint requiring grounding in retrieved context, a model asked something the data does not cover will construct an answer that sounds exactly as confident as a correct one.

It leaks across permission boundaries. A user asks a general question and the answer contains a detail from a document they should never have seen. The model did nothing wrong. The retrieval layer had no concept of who was asking.

Each of these gets diagnosed as “the model hallucinates”. Each is an architecture problem one layer down.

Where the work actually is

Across the agents we have built, the split of engineering effort looks like this:

Layer	Share of effort
Model selection and prompting	~15%
Retrieval quality	~50%
Guardrails and permissions	~35%

Retrieval dominates because it is where most of the real decisions live. What counts as a chunk. How to preserve the context a chunk was cut out of. Whether to filter by metadata before searching or rank after. How to encode that one document supersedes another. What to do when the top result is only weakly relevant — pass it through, or return nothing.

None of this is glamorous, and none of it appears in a demo.

The rule that makes agents usable

Every answer must cite the record it came from.

This sounds like a UI detail and is actually the constraint that changes the system’s character. If the agent must produce a citation, it cannot answer from parametric memory. If it cannot find a source, it must say so. And critically — the user gains a way to verify in one click, which means the answer no longer requires trust.

An unverifiable answer in a business context is close to worthless. Nobody is going to change a product, quote a client or make a clinical decision on the strength of something they cannot check.

The uncomfortable implication

If half the work is retrieval and a third is guardrails, then the quality of an AI system is mostly determined by the quality of the data layer beneath it.

Which means a company with scattered, stale or partially broken data does not have an AI problem. It has a data problem that an AI project has made visible.

The pilot did not fail. It reported something accurate about the underlying systems, and the report was unwelcome.

What to do instead

Fix the data layer as part of the AI project, not as a prerequisite to it — otherwise it never gets funded.

Pick one narrow question the agent must answer correctly, and make that the acceptance criterion instead of a general impression of capability.

Require citations from the first prototype, so the constraint shapes the architecture rather than getting bolted on when trust starts slipping.

And be suspicious of any pilot that goes unusually smoothly. It usually means the data was curated, and curation is the one thing that will not survive contact with production.