The framing#

Stop treating evals as research. Treat them as integration tests with fuzzy assertions. A good eval suite is:

  • Deterministic where it can be
  • Has clear pass/fail per case
  • Runs in CI
  • Owns its own dataset, versioned

Pass/fail, not score-only#

Scores drift. Booleans don't.