The development
What happened
Ai2 has released XYBench, an evaluation of whether language-model assistants help with a user's underlying goal instead of reinforcing a mistaken approach. Its reported experiments distinguish mentioning a useful solution from making that solution the focus.
Key facts
The 8,115-question collection mixes forum and curated examples with 6,272 questions generated from WikiHow guides. It is not entirely a record of naturally occurring user requests.
In single-turn tests of five models, underlying-goal advice was the main focus in 35–41% of Stack Overflow responses, versus 47–57% focused on the stated request, the researchers report.
Claude Sonnet 4.5 serves as automated judge. Two human reviewers assessed 500 responses; agreement on emphasis was lower than agreement on whether a solution appeared. Coverage is English and mainly procedural.
The context
Why it matters
TideReach assessment: an assistant can mention the right fix yet devote its answer to the wrong plan. Evaluating emphasis separately could reveal that failure. These author-reported, single-turn results do not establish performance in extended conversations or independently validate deployed assistants.