Google DeepMind Releases XYEval to Assess Agents' Compliance with Misleading Advice

Google DeepMind researchers Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, and Been Kim released a preprint paper, "XYEval: Agents say yes to bad advice," proposing a meta-evaluation framework called XYEval to assess how agents perform when faced with misleading user advice. The paper extends the "XY problem" to agent scenarios, where users ask about solution X they are attempting rather than the actual problem Y. XYEval preserves the original task environment and verifiable answers, only appending seemingly plausible but misleading user advice to the instructions. The misleading advice is generated by a generator after seeing the full task and is appended as a confident suggestion rather than a hard requirement, keeping the original task solvable. The evaluation covers six benchmarks—τ2-bench, SWE-bench Verified, SWE-bench Pro, Terminal-Bench, Humanity's Last Exam, and MCP-Atlas—and five models, with misleading advice for τ2-bench generated by rules set by human experts. The paper reports that task performance drops significantly after XY mutation, with a relative decline of up to 46.7%. On τ2-bench, performance declines more when users demand a detailed explanation before accepting a better solution. Simple system instructions encouraging models to be aware of the XY problem only partially mitigate the issue. Related code is released on the GitHub repository google-deepmind/xyeval.