The Problem
Every AI team faces the same question: "Which model should we use?" The typical answer is based on public leaderboards and a handful of demo prompts. But public benchmarks measure generic capabilities, and demo prompts test the center of the distribution.
Production traffic lives in the tails. The model that scores highest on MMLU might hallucinate on your specific domain. The model with the best HumanEval score might fail at multi-step reasoning in your workflow. You can't know until you test on your actual use cases at scale.
What I Built
MBPI is an automated platform for systematically finding failure modes across frontier models. I designed and built it solo.
Architecture
The platform orchestrates API calls across multiple model providers — OpenAI (GPT-5.1), Anthropic (Claude 4.5), Google (Gemini 3 Pro), Tencent (Hunyuan), and others. At peak, it processes 128K API calls per day, running structured test scenarios and collecting responses for automated evaluation.
LLM-as-a-Judge System
Human evaluation doesn't scale. At 128K calls/day, you'd need an army of evaluators. So I built an LLM-as-a-Judge system that processes 100K+ evaluations daily.
Getting automated judges to be reliable is hard. Naive implementations have well-known biases: position bias (preferring the first response), verbosity bias (preferring longer responses), and self-preference bias (GPT judges prefer GPT outputs). My system addresses all of these:
- Multi-judge consensus: 3 different model families evaluate each output
- Bidirectional comparison: Every pairwise judgment is run in both orders
- Rubric decomposition: Instead of one overall score, I score 4-6 specific dimensions
- Calibration set: 200 human-labeled examples verify judge accuracy weekly
The result: 0.89 Spearman correlation with expert human labels at roughly 1/100th the cost of full human evaluation.
Failure Taxonomy
After months of systematic testing, I've identified 9 categories where frontier models consistently fail:
- Instruction following under constraint stacking — Stack 5+ constraints and compliance drops below 60%
- Numerical reasoning with unit conversions — 15-30% error rate on multi-conversion problems
- Negation and absence — Error rates roughly double compared to positive versions of the same question
- Temporal reasoning — Especially with timezone conversions and DST transitions
- Self-consistency in long outputs — 25-40% of documents over 1,500 words contain internal contradictions
- Graceful scope boundaries — Models hallucinate instead of admitting they lack information
- Format compliance under pressure — JSON validity degrades with output length
- Multi-document synthesis — Models summarize individually instead of synthesizing across sources
- Calibrated uncertainty — Confidence doesn't correlate with actual accuracy
Why This Matters
When I run an Evaluation Sprint for a client, this infrastructure is what powers it. Instead of testing with 20 hand-picked prompts, I can test with thousands of structured scenarios and get reliable automated scores.
The output isn't "Model X is better than Model Y." It's "Model X fails at constraint stacking and temporal reasoning on your specific scenarios, while Model Y fails at format compliance and scope boundaries. Here's which one is better for your use case, and here are the guardrails you need for either."
That's the difference between evaluation and benchmarking-theater.