Esta edición aún no está disponible en español. Mostramos la edición en inglés.

A practitioner's honest account of building an evals pipeline at a 29-person edtech company with only 6 engineers — including the parts that failed. Lucas covers going from spreadsheets to a Claude + LangFuse workflow, discovering that inter-annotator agreement was worse than a coin flip, and eventually calibrating 4 LLM judges into production. If you're wondering what evals look like outside of course material, this is it.

• Initial inter-annotator agreement was worse than random for many criteria — the fix was months of rubric clarification and re-labeling, not better models.
• Biggest resource-saving mistake correction: stop labeling data for non-problems; only label failure modes actually discovered through error analysis.
• Calibrated judges serve double duty: CI gates on PRs to catch regressions, plus daily monitoring on ~80% of production output to track quality drift.
• They calibrated judges with GPT-4 while generating with a cheaper model — but once calibrated, you must keep the same judge model or your TPR/TNR metrics are invalid.

Parte de Edición Nº 002: How Anthropic ships with its own agents, evals in the trenches, and the new physics of small teams