The development

What happened

Ai2 has released XYBench, an evaluation of whether language-model assistants help with a user's underlying goal instead of reinforcing a mistaken approach. Its reported experiments distinguish mentioning a useful solution from making that solution the focus.

Key facts

  • The 8,115-question collection mixes forum and curated examples with 6,272 questions generated from WikiHow guides. It is not entirely a record of naturally occurring user requests.

  • In single-turn tests of five models, underlying-goal advice was the main focus in 35–41% of Stack Overflow responses, versus 47–57% focused on the stated request, the researchers report.

  • Claude Sonnet 4.5 serves as automated judge. Two human reviewers assessed 500 responses; agreement on emphasis was lower than agreement on whether a solution appeared. Coverage is English and mainly procedural.

The context

Why it matters

TideReach assessment: an assistant can mention the right fix yet devote its answer to the wrong plan. Evaluating emphasis separately could reveal that failure. These author-reported, single-turn results do not establish performance in extended conversations or independently validate deployed assistants.