A debugging benchmark fell from 86.2 to 25.9 and the internet called the model "caged". Only 3 of the 12 tasks reached it: a new safety classifier rerouted anything security-adjacent, and every reroute scored zero. Words like "vulnerability" and "fix" trip it. #AIEvaluation #AIAssurance

FutureSearch ran 150+ web research tasks on the top models; for three of the four, higher effort gave "equal or worse scores while costing significantly more". Anthropic says effort "can be miscalibrated in either direction" and advises an effort sweep on your own evals. #AIEvaluation #AIAssurance