An AI agent at a major airline once told a customer they could book a flight and claim a bereavement refund retroactively. The policy didn't exist. The model simply hallucinated it to be helpful. The airline lost the court case, but more importantly, they lost the trust of thousands of customers who watched the screenshot go viral.
The industry response to hallucinations has mostly been "better prompts" or "more fine-tuning." We think those are fragile mitigations for a structural problem. If a model can generate text without proving where the facts came from, it will eventually generate a lie. In support, a lie is a liability.
At Beamdesk, we solved this by making citations mandatory. Not as a UI ornament, but as a hard guardrail in the inference loop. If the model can't point to the specific Knowledge Base (KB) chunk that supports a claim, the reply is blocked before it ever touches a customer's screen.
The Problem: Confident Fabrication
LLMs are designed to be agreeable. When a customer asks, "Can I get a refund if my dog ate the charger?", a naive model sees the word "refund" and "charger" and tries to find a path to "yes." If the retrieved context is vague, the model fills the gaps with what sounds plausible based on its training data.
The danger isn't the obvious nonsense. It's the "almost right" answer. A model that says the return window is 30 days when it's actually 14 is worse than a model that starts reciting poetry. It looks like a legitimate business answer. It creates a promise the team has to either break (angering the customer) or keep (costing the business).
Most AI helpdesks treat Retrieval-Augmented Generation (RAG) as a suggestion. They provide the context and hope the model uses it. We treat RAG as a constraint. Every factual claim must be anchored to a source ID.
Implementation: The Citation Loop
We enforce this through a two-step verification process in our AI kernel. We don't just ask the model to "be honest." We structure the generation and then audit the output using `lib/ai/citation-format.ts` and `lib/ai/guarded-reply.ts`.
First, the prompt enforces a strict output schema. The model must wrap any policy claim, price, or procedure in a citation tag that references a specific document ID from the retrieved context.
// lib/ai/citation-format.ts
export type CitedClaim = {
text: string;
sourceId: string;
relevanceScore: number;
};
// Example model output:
"You can return your item within 14 days [ref:kb_842]
as long as the seal is unbroken [ref:kb_911]." Second, the `guarded-reply.ts` middleware intercepts the stream. It extracts every citation, verifies that the `sourceId` exists in the chunks actually provided to the model, and performs a secondary "relevance check." If a model tries to use a general shipping chunk to justify a refund claim, the validator sees the mismatch.
// lib/ai/guarded-reply.ts
async function validateCitations(draft: string, sources: KBChunk[]) {
const citations = extractCitations(draft);
for (const cit of citations) {
const source = sources.find(s => s.id === cit.sourceId);
if (!source) throw new CitationError("Invalid source ID");
const isSupported = await verifyGrounding(cit.text, source.content);
if (!isSupported) {
console.error("Hallucination detected: claim not supported by source");
return { status: 'block', reason: 'unsupported_claim' };
}
}
return { status: 'pass' };
}Quarantine for Sensitive Replies
Not all claims carry the same weight. A claim about the color of a shirt is low risk. A claim about a $5,000 refund or a legal liability exception is high risk. We use "policy quarantine" for sensitive intents.
When the router detects a high-stakes intent (Refund, Cancellation, Escalation, Policy Exception), it adds an extra layer of scrutiny. Even if the citations pass the automated check, if the confidence score on the grounding is below a certain threshold, the reply is held for human review.
This is where our "Honest Outcome" philosophy meets engineering. We would rather add 200ms of latency and a human review step than ship a perfectly formatted hallucination. A blocked reply is a temporary delay; a wrong reply is a permanent stain on the brand.
Honest Tradeoffs
Citations by default isn't free. There are three main costs we observed in our first six months of production:
- Latency: The secondary verification step and the complex prompting add roughly 150ms to 300ms to the total response time. In a world obsessed with "instant" AI, we chose to be "correct" instead.
- KB Coverage Requirement: This system fails fast. If your Knowledge Base is thin, the AI will constantly hit "I don't know" or block its own replies. You can't hide a bad KB behind a smart model when citations are mandatory.
- False Negatives: Occasionally, a model writes a perfectly valid answer but fails to format the citation correctly or picks a "kinda related" chunk instead of the "exactly right" one. Our validator blocks these. We call this a "safe fail."
What It Means For Buyers
When you look at an AI helpdesk, ask to see the audit trail for a refund request. If the answer is just a block of text, you are trusting the model's vibe. If the answer is a collection of cited claims, you are auditing the business's policy.
For technical buyers, this means Beamdesk is "auditable by design." Every reply in the ticket history has a `citations` metadata array. You can see exactly which version of which KB article was used to generate that specific sentence. If a policy changes, you can trace which AI answers were affected.
Hallucination guards aren't about making AI smarter. They are about making AI accountable. By enforcing citations at the kernel level, we turn the "black box" of LLM generation into a transparent, verifiable support operation.