Anthropic’s TASTE Benchmark Exposes Weak AI Research Judgment

Anthropic has introduced TASTE, a benchmark designed to test whether artificial intelligence models can evaluate research proposals and make judgments similar to those of experienced researchers. The assessment focuses on higher-order decisions such as identifying important problems and setting research priorities. Those skills are crucial for allocating resources in AI safety research and differ from the language generation and idea production tasks at which leading models often excel.

Results released by Anthropic show that the strongest model agreed with the expert benchmark only about 60% of the time, while most models performed near the level of random guessing. The findings suggest that current systems may generate plausible research ideas without reliably determining which questions deserve attention first. Human oversight therefore remains essential if AI is used to review proposals, direct research programs or shape safety priorities.