Evaluating RAG in Laravel: Is Your Agent Retrieving the Right Chunks?

· 7 min read
laravelairagpgvectorlaravel-ai-sdk
Evaluating RAG in Laravel — a query arrow pulling the wrong document, a red warranty card, out of a stack while the correct emerald shipping card stays put

You shipped a support agent last month. It reads your help docs and answers customer questions. The demo looked great. Returns, refunds, shipping, warranty: ask it anything and it pulls an answer from your knowledge base.

Then a customer asks “do you ship to Canada?” and it answers from your damaged-items policy. Confidently. Wrong.

No error. No exception. No failed test. Retrieval grabbed the wrong document, the model wrote a fluent paragraph on top of it, and nothing in your stack noticed.

Your tests are green. They were never checking the part that broke.

Every Laravel RAG tutorial stops at the demo

The official Ship AI with Laravel series builds a RAG agent. Tighten’s guide builds one. A dozen blog posts build one. They all stop at the same spot: the agent answers a question on screen, and the post ends.

None of them ask the next question. Is retrieval actually any good?

That question is the whole job. A RAG agent is two systems pretending to be one: a retriever that finds context, and a model that writes from it. The model is only ever as right as the chunks you hand it. Give it the warranty doc for a shipping question and it answers about warranties, in clean prose, with total confidence.

You can’t eyeball that. You measure it. This is the sequel to evaluating your AI output: that post graded a classifier’s answer, this one grades the layer underneath an agent. Did retrieval even fetch the right context before the model opened its mouth?

The system under test

I built this on the Laravel AI SDK with pgvector, in the same support-agent world as my testing and evals posts. Nine policy documents (returns, shipping, refunds, warranty, account, exchanges, damaged items, promotions, privacy), chunked, embedded with text-embedding-3-small, and stored in Postgres. The full project is on GitHub at MujahidAbbas/laravel-rag-evals: clone it, add an OpenAI key, and the evals run against real pgvector.

One rule comes before any eval, and it’s the one most people get wrong: ingestion and querying must use the same model and the same dimensions. Embed your chunks one way and your queries another, and every distance you compute is noise. So both paths go through a single class:

// app/Support/Embedder.php
class Embedder
{
    public function model(): string
    {
        return config('knowledge.embedding_model');
    }

    public function dimensions(): int
    {
        return config('knowledge.embedding_dimensions');
    }

    public function many(array $texts): array
    {
        $pending = Embeddings::for($texts);

        // Only the text-embedding-3-* family accepts a custom dimension count.
        if (str_starts_with($this->model(), 'text-embedding-3')) {
            $pending->dimensions($this->dimensions());
        }

        return $pending->generate(model: $this->model())->embeddings;
    }

    public function one(string $text): array
    {
        return $this->many([$text])[0];
    }

    public function toVector(array $embedding): string
    {
        return '['.implode(',', $embedding).']';
    }
}

Retrieval is one cosine query against pgvector, nearest first:

// app/Support/KnowledgeBaseRetriever.php
public function retrieve(string $query, ?int $k = null): Collection
{
    $k ??= config('knowledge.retrieval_k');

    $vector = $this->embedder->toVector($this->embedder->one($query));

    return collect(DB::select(
        'select id, source, chunk_index, content,
                1 - (embedding <=> ?::vector) as similarity
         from document_chunks
         order by embedding <=> ?::vector
         limit ?',
        [$vector, $vector, $k],
    ));
}

That’s the thing we’re testing. Now the test.

A golden set is fixtures for retrieval

Write down real questions and the document that should answer each one. Not invented questions. The ones your customers actually type.

[
  {
    "question": "Do you ship to Canada?",
    "expected_source": "shipping-policy.md",
    "expected_substring": "Canada, the United Kingdom"
  },
  {
    "question": "When will my refund show up on my card?",
    "expected_source": "refunds-billing.md",
    "expected_substring": "5 to 10 business days"
  },
  {
    "question": "Is there a fee for returning an item?",
    "expected_source": "returns-policy.md",
    "expected_substring": "15% restocking fee"
  }
]

Make it hard on purpose. A set where every question maps cleanly to one obvious document tells you nothing. You want the cases that sit on the line: a refund question that could be returns or billing, a broken item that could be returns or warranty or damaged-items. Failures hide on the line, never in the middle. My set has 32 of these, with deliberate topic overlap between documents.

Notice each case carries two labels. expected_source is the document. expected_substring is the exact sentence that answers the question. You’re about to see why one isn’t enough.

Measure the right thing, because the document-level number lies

Here’s the eval. No fake() anywhere: it embeds every question against the real OpenAI API and queries the real pgvector database, because the whole point is to see what retrieval actually does.

// tests/Evals/RetrievalEvalTest.php
it('retrieves the correct context for support questions', function () {
    $golden = json_decode(
        file_get_contents(base_path('tests/Evals/golden/support-questions.json')),
        true,
    );

    $retriever = app(KnowledgeBaseRetriever::class);
    $total = count($golden);
    $sourceHit1 = 0;
    $answerHit1 = 0;

    foreach ($golden as $case) {
        $chunks = $retriever->retrieve($case['question'], 3);
        $top = $chunks->first();

        if ($top?->source === $case['expected_source']) {
            $sourceHit1++;
        }

        if ($top && str_contains($top->content, $case['expected_substring'])) {
            $answerHit1++;
        }
    }

    $this->assertGreaterThanOrEqual(0.85, $sourceHit1 / $total);
    $this->assertGreaterThanOrEqual(0.80, $answerHit1 / $total);
});

Two metrics, on purpose. source hit@1 asks whether the nearest chunk came from the right document. answer@1 asks whether that nearest chunk actually contained the sentence with the answer.

Run it:

php artisan test --testsuite=Evals
source hit@1: 94% (30/32)
source hit@3: 100% (32/32)
answer@1:     84% (27/32)

Stop on those two numbers. Source-level retrieval looks 94% healthy. Measured at the passage, it’s 84%. In five cases the search found the right document but ranked a chunk that didn’t hold the answer, so the model got a paragraph about the right topic without the fact it needed.

That ten-point gap is the most expensive thing in this whole post, and almost every RAG eval guide misses it, because they stop counting at the document. Your retriever can look fine and still be quietly feeding the model the wrong sentence. You only see it when you measure the passage.

The regression that ships without an error

Now watch the eval earn its keep.

pgvector storage grows with your dimension count. Someone on your team, six months from now, decides to shrink the index to save space and reads that text-embedding-3-small lets you ask for fewer dimensions. Reasonable. They drop it from 1536 to 16 and redeploy.

KB_EMBEDDING_DIMENSIONS=16 php artisan migrate:fresh --force
KB_EMBEDDING_DIMENSIONS=16 php artisan kb:ingest
KB_EMBEDDING_DIMENSIONS=16 php artisan test --testsuite=Evals
source hit@1: 47% (15/32)

Wrong top result for:
  "Do you ship to Canada?"
    expected shipping-policy.md, top result was damaged-or-defective.md
  "How do I reset my password?"
    expected account.md, top result was warranty.md
  "How much does express shipping cost?"
    expected shipping-policy.md, top result was refunds-billing.md

There it is. The Canada question now pulls the damaged-items policy. The password question pulls the warranty doc. Without the eval, this reaches production silently. The app boots. Every page loads. The agent answers every question in fluent, confident English, and more than half of those answers come from the wrong document. The first time you hear about it is from a customer, or worse, you never do.

With the eval, the build goes red before the change merges, and it tells you exactly which questions broke. That is the difference between a config tweak and an incident.

The judge is also an AI, so check it before you trust it

Retrieval metrics grade the context. They don’t grade the answer. For that you need a second model to read the agent’s reply and judge it. So I built one:

// app/Ai/Agents/ResponseJudge.php
class ResponseJudge implements Agent, HasStructuredOutput
{
    use Promptable;

    public function instructions(): Stringable|string
    {
        return <<<'PROMPT'
            You grade a customer support reply for faithfulness to company policy.
            You are given the question, the official policy text, and the reply.
            The policy text is the single source of truth.

            Pass the reply if it answers the question and every claim it makes is
            supported by the policy text. Fail it only if it contradicts the policy,
            does not answer the question, or invents a detail the policy does not support.

            Return a verdict, a 1-to-10 score, and one sentence of reasoning.
            PROMPT;
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'verdict' => $schema->string()->enum(['pass', 'fail'])->required(),
            'score' => $schema->integer()->min(1)->max(10)->required(),
            'reasoning' => $schema->string()->required(),
        ];
    }
}

The eval runs the full agent on each question, then feeds the reply to the judge:

// tests/Evals/AnswerEvalTest.php
foreach ($golden as $case) {
    $reply = (new SupportAgent)->prompt($case['question'])->text;

    $policy = file_get_contents(database_path('knowledge/'.$case['expected_source']));

    $verdict = (new ResponseJudge)->prompt(<<<PROMPT
        Question: {$case['question']}

        Official policy text (the single source of truth):
        {$policy}

        Support agent reply:
        {$reply}
        PROMPT);

    $verdict['verdict'] === 'pass' ? $passed++ : $failures[] = $case['question'];
}

My first run scored 28% pass. I almost wrote the agent off as a disaster.

Then I read the judge’s reasoning. It was failing answers for “adding unsupported details like a 30-day return window and a 15% restocking fee.” But those details are in the knowledge base. The agent was right. The judge was wrong.

The bug was the reference. My first version handed the judge a single expected_substring as the only ground truth and told it to reject anything beyond that one sentence. So it punished the agent for being thorough. The moment I gave the judge the full policy document as ground truth, the score jumped to 94%.

Read that again, because it’s the part people skip. The first thing my eval caught was a bug in my eval. A judge is a model you also haven’t checked. Calibrate it against your own judgment on a handful of cases before you believe a single number it gives you. If you wire up an LLM judge, see a percentage, and ship on it, you’re trusting one unverified model to grade another.

Make it a gate, not a vibe

These evals cost money and hit the network, so they don’t belong in the suite that runs on every commit. They go in their own suite, run on a schedule and before any deploy that touches the embedding model, the dimensions, or the chunking:

php artisan test --testsuite=Evals

One Laravel-specific trap on the way there. Your phpunit.xml almost certainly forces DB_CONNECTION=sqlite and an in-memory database for tests. That database has no pgvector extension and none of your embedded chunks. The eval suite has to point back at the real Postgres database, which you can do for that suite alone in tests/Pest.php:

pest()->extend(TestCase::class)
    ->beforeEach(function () {
        config([
            'database.default' => 'pgsql',
            'database.connections.pgsql.database' => 'laravel_rag_evals',
        ]);

        DB::purge('pgsql');
    })
    ->in('Evals');

Skip this and your eval silently runs against an empty sqlite database and passes for all the wrong reasons. Which would be its own kind of false confidence.

What you can do now

Before today, you changed your chunking or your embedding model or your dimensions and you crossed your fingers. You found out it broke from a customer asking about Canada and hearing about damaged goods.

Now you change one of those knobs and a number tells you. You can shrink your dimensions to cut your bill and prove retrieval still holds, or learn in CI that it didn’t. You can measure not just whether the right document came back, but whether the right sentence did. And you know not to trust your judge until you’ve judged the judge.

Your demo proved the agent can answer. Your evals prove it answers from the right page. Ship the next change knowing which one you did.

The whole project is on GitHub: MujahidAbbas/laravel-rag-evals. Clone it, point it at your own docs, and find out what your retrieval is really doing.

All articles