My RAG system answers thousands of search queries every day.
I've personally read maybe 200 of those answers.
That used to keep me up at night.
Because when you ship an AI product, the demo always works.
You try five queries. The answers look great. Everyone nods.
Then real traffic arrives and the real question shows up:
Is this system actually good… or did we just get lucky with five examples?
Here's the problem: reading more responses doesn't solve this.
Reading 10,000 outputs isn't evaluation. It's a slow way to burn out a smart engineer.
So I built an LLM judge.
The first version was useless.
Like everyone else, I started with:
Rate this answer from 1–10.
Huge mistake.
Numeric scores from LLMs are noise dressed up as signal.
The same answer gets a 6 today and an 8 tomorrow. A "7" means nothing unless the model has a precise definition of quality.
Most of the time, you're just averaging vibes.
What actually worked:
- Binary verdicts instead of scales
- Reasoning before scoring
- Explicit evaluation criteria
- Tight prompts preventing invented requirements
The binary change mattered most.
Instead of "rate the answer," I asked:
Does this product match every requirement in the query — yes or no?
Suddenly evaluations became reproducible.
Then I hit a subtler failure mode.
A user searched for "blue armchair." The catalog had a perfect blue armchair.
The judge rejected it.
Why? Because the model decided the user "probably meant fabric upholstery." It invented a requirement that never existed.
One prompt line fixed it:
Do not reject a product for any reason not explicitly stated in the query.
That single sentence dramatically improved accuracy.
But it also taught me the real lesson about LLM judges:
The judge is just another model.
It has biases. It prefers certain answer styles. It over-indexes on confidence. It says "yes" too easily.
You don't eliminate bias in an LLM judge. You choose which bias you want.
So I treated the judge like any other model: build eval sets, calibrate against humans, measure agreement rates.
And that's when something clicked for me.
The judge doesn't replace human review. It decides what humans should review.
My evaluation harness runs thousands of queries, scores everything automatically, then surfaces the failures most likely to matter.
Out of 10,000 evaluations… I read five.
And they're usually the right five.
That's the entire trick.
The judge isn't a grader. It's a router for human attention.
If your LLM quality process is still "we spot-check outputs manually" — what happens the day a prompt change quietly drops accuracy by 8%, and nobody opens the five examples that would've caught it?