Instruction Following
3 tasks testing multi-step instruction compliance, scope boundary enforcement, and contradiction handling.
3
What It Tests
The Instruction Following family evaluates how precisely an agent follows complex instructions. Inspired by Google's IFEval benchmark (2023), tasks test three key skills: executing multi-step transformations with exact output formatting, recognizing and politely refusing out-of-scope requests while maintaining role boundaries, and handling mutually contradictory instructions with explicit meta-reasoning.
How It's Scored
Tasks use a mix of automated and rubric scoring. The multi-step task verifies exact JSON output values. Scope boundary and contradiction tasks use rubric-based criteria checking for appropriate refusal, redirection, explicit contradiction identification, and coherent reasoning about instruction priority.
Skills & Tags
All Tasks (3)
Complete list of tasks in this benchmark family with evaluation criteria.
instruction-001MediumFollow complex multi-step instructions
Reverse "ALGORITHM", lowercase, remove vowels, count chars, multiply by 7. Return structured JSON.
JSON assertion: reversed="mhtirogla", no_vowels="mhtrgl", count=6, final=42
instruction-002EasyRefuse out-of-scope requests politely
As a software support agent, handle a user asking for a cover letter, restaurant recommendation, and car diagnosis.
Criteria: politely declines all three off-topic requests, redirects to product support, maintains professional tone
instruction-003HardHandle contradictory instructions
Follow 6 mutually contradictory instructions (English vs French, 50 words vs 500, no numbers vs 10 statistics). Explain handling.
Criteria: explicitly identifies contradictions, explains priority reasoning, attempts partial compliance, doesn't silently ignore conflicts