Anthropic has tested whether AI models can judge which AI safety research ideas are worth pursuing. Its new TASTE benchmark compares model decisions with the preferences of experienced AI safety researchers. The strongest model in the standard setup reached 60% agreement with the benchmark labels, while Anthropic estimated human agreement at 77%.
That gap is only part of the story. The more interesting finding is how much work Anthropic had to do before it could even call human judgments a benchmark. The researchers evaluated 92 pairs of research proposals, first making their choices independently and then discussing cases where they disagreed.
Before those discussions, estimated agreement for strong-confidence preferences was 53%. After discussion, it rose to 68%, and Anthropic applied further filtering to produce the final set of labels, with an estimated human agreement of 77%. The human answer key was not simply collected; it had to be worked out through review.
Dhara Patadia, an AI Specialist and Product Innovator who works on machine-learning automation, says this is a common problem in AI development. “When there is a fixed answer, evaluation is relatively straightforward. The harder cases are the ones where trained people can look at the same input and come to different conclusions.”
“Before you ask whether a model got the answer right, you need to know how stable your definition of ‘right’ is,” Dhara says. That problem often appears much earlier in the ML pipeline, during data annotation. Dhara has previously written about inter-annotator agreement, which measures how consistently different annotators label the same data.
Low agreement can point to unclear instructions or cases that the labelling schema does not cover well. The TASTE experiment illustrates a similar problem in a more subjective setting, where researchers assess the quality of proposed AI safety experiments. In this kind of task, disagreement can reveal something important about the evaluation process itself.
Anthropic found that researchers disagreed for different reasons. In some cases, they held different views about whether a proposed method would work. In others, one person had misunderstood part of the proposal, which means not every disagreement should be treated in the same way.
“If two people disagree because the instructions are vague, that can often be fixed,” Dhara says. “If they understand the task in the same way and still disagree, then the ambiguity is part of the task itself. You shouldn’t hide those two situations inside the same accuracy number.”
This becomes more relevant as companies automate more of the work around model training and evaluation. Models can already handle repetitive steps that once required manual labelling, and they can route uncertain cases for review. The difficulty starts when the system is expected to make a judgment for which the human reviewers have not settled on a common standard.
A confidence score does not solve that problem. A model can be highly confident while applying an inconsistent rule, especially if that inconsistency already exists in the training data. “The useful question isn’t simply whether a model can produce an answer,” Dhara says.
“It is whether the process is defined well enough for that answer to be useful,” she adds. “If the human reviewers keep disagreeing, that is information about the task. It shouldn’t automatically be treated as noise.”
The TASTE results also show why benchmark scores need context. A score of 60% can look poor next to 77% human agreement, but the 77% figure came after researchers discussed disagreements and Anthropic filtered the resulting judgments. The benchmark itself contains a human quality-control process that is easy to miss when results are reduced to a single number.
“Human feedback is often treated as the reference point, but human judgments need checking too,” Dhara says. “For subjective tasks, I would want to know how the labels were created before drawing strong conclusions from the model score.” That question becomes more important as AI systems are evaluated on work that may have several defensible answers.
Anthropic is careful about the limits of TASTE. The benchmark is small, and the researchers say its confidence intervals are too wide to support precise rankings between models. Its more useful contribution may be showing how much human work still goes into establishing the standard against which an AI system is judged.
The challenge is not always getting a model to imitate a human decision. Sometimes the difficult part comes earlier, when the humans themselves have to decide what a good answer looks like. Until that standard is stable, automating the judgment does not remove the uncertainty behind it.
