- Published on
Tests check whether code does what it was written to do. They have nothing to say about whether a judgement was any good. Scoring a non-deterministic system means comparing recorded predictions against observed outcomes, and most systems already emit both without recognising that they have the raw material for an eval harness. Why calibration matters more than accuracy, what bucketing is for, and the two problems - outcome lag and correlated error - that make this harder than it looks.