Your retrieval eval passes. answer@1 sits at 84%, the build is green, and you ship.
Then a customer types “these shoes are too small, can I get the next size up without paying again?” and the agent answers from your returns policy instead of exchanges. Right topic. Wrong page. Fluent and confident and wrong.
The eval never caught it, because the eval never asked a question that messy.
This is part three of the RAG series. Part one built the pipeline. Part two measured it. This one fixes the gap the measuring exposed: the search finds the right document and still ranks the wrong chunk first. The fix is a reranker, and you can build it from the model you already pay for.
Your eval set was too polite
The golden set in part two reused each document’s own words. “Is there a fee for returning an item?” maps straight onto the returns policy, because both say “return” and “fee.” That set proved the pipeline works. It also flattered it.
Real customers don’t talk like your docs. They say “send it back,” not “return.” They say “my money still isn’t back,” not “refund.” They describe a situation instead of naming the topic. When the words in the question and the words in the answer drift apart, single-vector search starts guessing.
So I wrote a second fixture: ten deliberately messy questions, each one phrased to avoid the answer document’s vocabulary.
[
{ "question": "These shoes are too small, can I get the next size up without paying again?",
"expected_source": "exchanges.md", "expected_substring": "Exchanges ship free" },
{ "question": "It was busted when I opened the box, is that a warranty thing?",
"expected_source": "damaged-or-defective.md", "expected_substring": "damaged-on-arrival case" },
{ "question": "I scored a deal with a code at checkout, what's my refund if I send it back?",
"expected_source": "promotions-and-gift-cards.md", "expected_substring": "amount you actually paid" }
]Look at the second one. The words “warranty thing” point straight at the warranty doc, but the answer lives in the damaged-items policy. The question actively misleads the search.
Run the same retrieval from part two against both sets and the gap opens up:
fair set hard set
hit@1 / hit@3 / answer@1
baseline 94 / 100 / 84 80 / 90 / 50Read the hard column. The right document lands in the top three 90% of the time, so retrieval mostly knows where to look. But answer@1 is 50%. On half the messy questions, the chunk ranked first does not contain the answer. The model gets a paragraph from roughly the right place and writes a confident reply on top of the wrong sentence.
That is the bug. Not “retrieval misses the document.” Retrieval finds the document and buries the answer at rank four.
Reranking is a second opinion on the top results
Cosine distance is one number per chunk. It measures how close two embeddings sit, which is a decent proxy for “related” and a poor proxy for “this exact passage answers the question.” Close and correct are not the same thing.
A reranker is a second pass. You pull a wide net of candidates with cheap vector search, then ask a smarter, slower judge to read each candidate against the actual question and reorder them. The judge sees the query and the passage together, so it can tell “mentions returns” apart from “states the return window.”
The Laravel AI SDK ships a native reranker for this, Reranking::of($docs)->rerank($query), backed by dedicated providers like Cohere and Jina. If you already pay one of those, it’s a single call. I didn’t want a second vendor and a second key for a support bot, so I reranked with the model already answering the questions.
The judge is one structured-output agent. Same pattern as the response judge from part two, different job: it reads the candidates and returns their order.
// app/Ai/Agents/ChunkRanker.php
class ChunkRanker implements Agent, HasStructuredOutput
{
use Promptable;
public function instructions(): Stringable|string
{
return <<<'PROMPT'
You rerank candidate knowledge-base passages by how well each one
ANSWERS the customer's question — not by how many words it shares
with the question. A passage that directly contains the answer ranks
above one that is merely on the same topic.
You are given the question and a list of candidate passages, each
tagged with a number like [0], [1], [2]. Return every candidate's
number in a single ordered list, most relevant first. Include all of
the numbers exactly once.
PROMPT;
}
public function schema(JsonSchema $schema): array
{
return [
'ranking' => $schema->array()->items($schema->integer())->required(),
];
}
}The schema does the heavy lifting. You force the model to return an array of integers, the candidate indexes in relevance order, so you never parse prose. One call ranks the whole pool at once.
The retriever wraps it around the cosine search from part two. Stage one casts the net, stage two reorders.
// app/Support/RerankRetriever.php
class RerankRetriever
{
private int $candidatePool = 12;
public function __construct(private KnowledgeBaseRetriever $retriever) {}
public function retrieve(string $query, ?int $k = null): Collection
{
$k ??= config('knowledge.retrieval_k');
// Stage 1: cast a wide net with plain cosine vector search.
$candidates = $this->retriever->retrieve($query, $this->candidatePool)->values();
if ($candidates->count() <= $k) {
return $candidates->take($k);
}
// Stage 2: number the candidates, let the model reorder them.
$numbered = $candidates
->map(fn ($c, $i) => "[{$i}] ({$c->source})\n".trim($c->content))
->implode("\n\n");
$response = (new ChunkRanker)->prompt(<<<PROMPT
Question: {$query}
Candidate passages:
{$numbered}
PROMPT);
$order = collect($response['ranking'] ?? [])
->filter(fn ($i) => is_int($i) && $candidates->has($i))
->unique()
->values();
// Anything the model dropped keeps its vector order at the tail, so a
// forgetful rerank never loses a chunk outright.
$ordered = $order
->map(fn ($i) => $candidates[$i])
->concat($candidates->reject(fn ($c, $i) => $order->contains($i))->values());
return $ordered->take($k)->values();
}
}Two details earn their keep. The filter drops any index the model hallucinates, so a bad rank never crashes the request. The concat re-appends anything the model forgot, in its original cosine order, so the pool can only get reordered, never shrunk.
It shares the exact retrieve(string, ?int) signature as the plain retriever, so it drops into the eval as another strategy with no other changes.
The number that justifies the whole post
Run both retrievers across both fixtures:
fair set hard set
hit@1 / hit@3 / answer@1
baseline 94 / 100 / 84 80 / 90 / 50
with reranking 97 / 100 / 97 100 / 100 / 100The hard set’s answer@1 goes from 50 to 100. Every messy question now gets the answer-bearing chunk ranked first. The “shoes too small” question stops landing on returns. The “busted out of the box” question stops landing on warranty.
The fair set barely moves, which is the point. It was already near the ceiling, and reranking didn’t drag it down. A new stage that helps your hard cases and leaves your easy ones alone is a stage you keep.
So that’s the whole sell. One structured-output agent, one wrapper, and the passage-level miss rate on messy questions drops to zero.
Now the part most reranking posts leave out.
Reranking reorders. It does not retrieve.
A reranker can only sort the candidates stage one handed it. If the answer chunk never made the net, no amount of reordering invents it. So the obvious worry: maybe reranking only looks this good because the net was huge.
I pulled twelve candidates out of a knowledge base that holds twenty-two chunks total. More than half the corpus goes into every rerank. Of course the answer is in there.
So I starved it. I shrank the pool and reran the hard set:
hit@1 / hit@3 / answer@1
baseline 80 / 90 / 50
rerank pool=4 100 / 100 / 100
rerank pool=8 100 / 100 / 100
rerank pool=12 100 / 100 / 100Even at four candidates, reranking still hits 100%. The gap never reopens. On this corpus you cannot starve it, because with two or three chunks per document the answer is always in the top four anyway. The reranker had the right chunk in hand every time and just promoted it.
That result is honest about exactly one thing: this knowledge base is too small to break. Twenty-two chunks is a demo, not a corpus.
Scale it to ten thousand chunks and the picture changes. A vocabulary-mismatched query like “send it back” can rank the entire exchanges document below the top fifty. Now your candidate pool never holds the answer at all. Reranking sorts fifty wrong chunks into a tidy order. The right one is still sitting at rank two hundred, unseen.
Reranking hits a wall there. When the answer chunk is in the pool, it’s the cheapest fix you have. When the query is so far off that the answer never makes the pool, reordering is useless, and you have to fix the query itself before it ever touches the index. Rewriting the question, generating a hypothetical answer to search with, fanning out across phrasings. That is the next post.
What it costs
Reranking with a chat model is not free. Each query now waits on an extra model call that reads a dozen passages before the agent can answer, a few seconds in my runs. A dedicated reranker like Cohere or Jina returns in milliseconds, which is the reason those services exist. You are trading latency for one less vendor. For a support bot answering one question at a time, the trade is fine. For a high-traffic search box, reach for the native Reranking call and a provider built for it.
And the 100% is this knowledge base, not a promise. Ten hard questions is enough to prove the stage works and nowhere near a benchmark. Run it against your own messy questions before you believe a number.
What you can do now
Part two gave you a number for retrieval. This gives you a lever to move it.
Write the messy fixture, the one phrased the way customers actually type, not the way your docs read. Add a rerank stage that pulls a wide net and reorders it with a model you already have. Watch answer@1 climb on the questions that used to land on the wrong page. And know the stage’s ceiling, so you reach for query rewriting when the answer stops making the pool, not before.
The whole project is on GitHub at MujahidAbbas/laravel-rag-evals. The reranker, the hard fixture, and the eval that prints these numbers are all in it. Clone it, point it at your own docs, and find the questions your search ranks fourth.