Twenty good cases beat two hundred borrowed ones. How to build evals you will actually maintain.
Write cases from your own failures
Every evaluation case should trace back to something that went wrong, or
nearly did, in real use. Borrowed benchmarks measure somebody else's risk.
One assertion per case
A case that checks three things tells you nothing when it fails. Split it.
Version the grader with the data
When the pass condition changes, old scores stop meaning the same thing.
Record which grader produced which number.