“It feels better” is not an evaluation. It is a story you tell yourself before a customer finds a regression. The most reliable way I have found to ship LLM upgrades safely is a five-axis matrix with hard numerical gates.
Why vibes-eval fails
Manual spot-checks are biased by recency and novelty. A new model that phrases things more confidently reads as smarter — even when its factual accuracy dropped. You need a harness that answers a single question: is this checkpoint safe to promote?
The five axes
- Reasoning — multi-step arithmetic, deduction, tool selection.
- Adversarial — prompt injections, jailbreaks, misleading context.
- Hallucination — closed-book questions with a known answer set.
- Domain fit — real user queries from production traces, hand-labeled.
- Latency — P50/P95 wall-clock at target concurrency.
The harness
Every axis is a dataset of (input, expected, grader) triples. Graders are either deterministic (regex, JSON schema, exact match) or an LLM judge with a rubric. Never mix the two on the same axis — you cannot compare scores across grading methods.
@dataclass
class Case:
axis: str
prompt: str
expected: Any
grader: Callable[[str, Any], float] # -> 0..1
def run(model, cases):
return {
axis: mean(c.grader(model(c.prompt), c.expected)
for c in cases if c.axis == axis)
for axis in AXES
}Regression gates
A candidate model must clear every axis by ≥ the baseline − 2%. A gain on reasoning does not excuse a hallucination regression. Ship the matrix in CI; block the deploy if any axis regresses.
Results
On the last rollout I ran, this matrix caught two regressions that human review missed — both hallucination spikes on domain-specific numerical claims. The final promoted model delivered ~30% aggregate quality lift while holding latency flat.