LLM judges agree on rankings and put the share of acceptable work anywhere from 3% to 98%

Harry Lyu and Neil Thompson audit LLM judges, the models asked to grade whether an AI’s output meets a workplace requirement. Their suite, O*NET-BENCH, is built from an existing survey of 45,796 worker ratings. They evaluate 33 existing judge configurations from six model families on 4,501 test ratings.

Twenty-five configurations reach a tie-aware pair accuracy of at least 0.60, although a TF-IDF baseline fitted on the training data nearly matches the strongest judge. On levels the judges scatter. They estimate that between 3.0% and 97.9% of responses are acceptable, against 61.1% for workers in the matching occupation.

In one fine-tuned lineage, a switch from pointwise scoring to a few-shot listwise protocol improved the ordering and reduced agreement with worker means at the task and occupation levels. The reversal replicated on a validation split with different tasks and workers, under criteria set in advance. Cross-validated calibration largely removes mean bias, but calibrated scores explain at most 8.5% of the variance in individual worker ratings.

The authors conclude that judges should be validated against the acceptance rates and aggregates their scores will be used to estimate. The abstract does not name the models.