Research & Receipts
Every claim made anywhere on this site, with a link to the primary evidence: published papers, merged pull requests, and case studies. If something can't be linked, it isn't listed here.
Published benchmark work
CL-bench: a context-learning benchmark arXiv:2602.03587
1,899 expert-authored tasks with 31,607 verification rubrics. Contributed evaluation tasks across Gaming, Coding, and Mathematics; served as strategic project lead on the delivery engagement.
CL-bench Life arXiv:2604.27043
405 human-curated context-task pairs and 5,348 rubrics grounded in messy real-life context. The strongest of ten frontier models solves only 19.3% of its tasks.
Open-source contributions to evaluation frameworks
inspect_ai (UK AI Safety Institute, 2k★)
5 merged PRs, hand-merged by JJ Allaire
lm-evaluation-harness (EleutherAI, 12.5k★)
Fail-fast guard preventing a silent ~17× accuracy regression on MoE models
- #3734 — data_parallel × expert_parallel incompatibility guard
Open-source tools
judge-calibration
Zero-dependency Python toolkit that measures whether an LLM-as-a-judge agrees with human labels, and catches the four biases a good correlation score hides: position, verbosity, self-preference, and sycophancy. Exits non-zero so it can gate CI directly.
Case studies
Writing
- The AI Agent Evaluation Handbook — Eight chapters mapping the whole discipline
- AI Evaluation Questions, Answered — Technical answers with measured numbers
- How to Benchmark Tool-Use in AI Agents — The 43-tool MCP harness, in full
- LLM-as-a-Judge: When It Works and When It Lies — Judge bias, calibration, rubric design
- What Breaks Frontier Models — Systematic adversarial testing
Talks & podcasts
None yet. If you run a podcast or meetup covering AI evaluation, agent benchmarking, or what actually breaks frontier models, get in touch — I'll bring real numbers rather than opinions.