When we first launched Beamdesk, we did what everyone does: we embedded our customers' Knowledge Bases into a vector database and used cosine similarity to find relevant chunks. It worked well enough for demos. But in production, "well enough" is where hallucinations start.
The problem with naive vector search is that it returns "kinda relevant" material. If a customer asks about a "missing power cord," a vector search might return articles about "missing items," "power requirements," and "cord safety." The model then has to sift through these chunks to find the specific policy for a missing accessory. This wastes tokens, increases latency, and—most critically—injects noise that can lead the model astray.
Six months ago, we switched to a dual-stage retrieval pipeline using Voyage Rerank-3. The results were immediate and measurable.
The Retrieval Gap
Embedding models are great at finding "neighborhoods" of meaning, but they are blunt instruments for precision. They compress a complex paragraph into a single list of numbers (a vector). In that compression, the nuance that separates a "standard return" from an "oversized freight return" can get lost.
In a helpdesk, retrieval quality isn't just about finding the right article. It's about finding the *only* chunk that matters. Every extra chunk you send to an LLM increases the chance it will hallucinate a hybrid policy or ignore a crucial exception hidden in the noise.
Enter Voyage Rerank-3
Voyage Rerank-3 is a cross-encoder. Unlike embedding models, which look at documents in isolation, a cross-encoder looks at the query and the document together. It's slower and more expensive than a vector lookup, but it is orders of magnitude more precise.
Our production pipeline now looks like this:
- Stage 1 (Recall): Perform a fast vector search across the entire KB to find the top 20 candidate chunks.
- Stage 2 (Precision): Pass those 20 chunks + the original query to Voyage Rerank-3.
- Selection: Only the top 3-5 reranked chunks are sent to the LLM to draft the response.
retrieval_log:
query: "can i return an opened mattress if it's too soft?"
vector_top_1: "Mattress Care Guide" (score: 0.82)
vector_top_2: "Returns Policy - General" (score: 0.81)
vector_top_3: "Opened Item Exceptions" (score: 0.79)
rerank_top_1: "Opened Item Exceptions" (score: 0.94)
rerank_top_2: "Mattress Trial Policy" (score: 0.91)
rerank_top_3: "Returns Policy - General" (score: 0.45)In this real-world example, the vector search liked the general "Care Guide" because it had the most mentions of "mattress." The reranker correctly identified that the "Opened Item Exceptions" and "Mattress Trial Policy" were the only ones that actually answered the user's specific problem.
The Impact: +40% MRR (Mean Reciprocal Rank)
We measured the impact of this change across several high-volume workspaces. The most telling metric was MRR (Mean Reciprocal Rank)—a measure of how often the "correct" answer is the very first thing the model sees.
With naive vector search, the correct chunk was in the top position roughly 58% of the time. With Voyage Rerank-3, that jumped to 82%.
For the LLM, this is the difference between having a clear set of instructions and having to dig through a messy desk. It leads to shorter prompts, fewer tool calls, and a dramatic reduction in "grounding low" flags from our Watchtower QA system.
Cost vs. Value
Voyage Rerank costs roughly $0.05 per 1k queries. For a typical helpdesk, this is a rounding error. Compared to the cost of an Opus or GPT-5 call—especially one that is bloated with 10 unnecessary KB chunks—reranking is actually a cost-saving measure.
By reducing the number of chunks we send to the LLM from 10 to 3, we often save more in input tokens than we spend on the rerank call itself.
The Honest Take
Reranking is not a magic fix for a broken Knowledge Base. If your articles are badly written, poorly chunked, or out of date, a reranker will just help you find the best of the bad options.
You still need a solid data strategy: clean titles, semantic chunking (splitting by intent rather than character count), and regular "KB freshness" audits. But once your data is clean, a dual-stage retrieval pipeline is the single most effective thing you can do to improve the "IQ" of your AI agent.
At Beamdesk, we've made Voyage Rerank-3 the default for all Pro and Enterprise workspaces. Because in support, the second-best answer is usually the wrong one.