xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

1ByteDance Seed    2M-A-P    3TokenWave.AI

  • What is xDailyBench： It covers 56 everyday scenarios and 248 authentic, open-ended tasks spanning personal life, work, and study, evaluating end-to-end agent performance on real-world everyday needs under realistic user constraints.
  • Why xDailyBench： We address the limitations of existing exam-style and professional workflow benchmarks, which often fail to capture ambiguous, personalized, context-dependent requests with no single ground-truth answer.
  • How xDailyBench is Built： It integrates real-user tasks, criterion-based binary rubrics covering both explicit and implicit requirements, and a unified taxonomy for evaluating agent capabilities.
  • What Insight We Find： Evaluation across 11 frontier models shows that the best-performing model achieves only 75.6% task-level performance, while all models exhibit over 9 percentage-point gaps on implicit requirements, highlighting intent inference as a persistent bottleneck for reliable task completion.
Daily-life task collage Correctness Sufficiency Compliance Harmlessness Presentation
arXiv Paper · Coming Soon Hugging Face Dataset · Coming Soon

Construction Pipeline

The construction of xDailyBench follows a rigorous, multi-stage pipeline that integrates real-user contributions, AI-assisted review, and iterative human quality control. Tasks are contributed by participants with practical experience using AI tools recruited through a crowdsourcing platform, who submit scenarios identical or similar to those they genuinely encounter in their personal lives, professional workflows, or studies.

Four-stage xDailyBench construction workflow from user-sourced task collection to pilot benchmark validation

Data Statistics

The benchmark spans broad daily-use scenarios and attaches a structured rubric and capability labels to every task.

248Expert-curated tasks
56L2 scenario categories
13.3Mean rubric criteria / task
5Capability dimensions
Nested distribution of 248 xDailyBench tasks across four primary domains and fine-grained scenarios
89 35.9%
Personal Life
82 33.1%
White-Collar Work
48 19.4%
Study & Research
29 11.7%
Other Cross-Domain Needs

Leaderboard

Eleven frontier models are compared under identical task instructions, tool access, and execution budgets; switch views to separate end-to-end task quality from requirement-level compliance.

Overall Success Rate Average score (%)
Higher is better
RankModelTask-levelExplicitImplicitGap

Capability Profiling

Each rubric criterion can be aggregated by capability dimension, turning a flat benchmark score into a model profile that exposes where failures concentrate.

Capability DimensionSeed-2.1-EvolvingKimi-K3GPT-5.6-solClaude-Opus-4.7
Info Understanding76.977.174.571.7
Instruction Following78.476.076.675.8
Analytical Reasoning73.673.673.473.4
Quantitative Reasoning74.772.662.662.2
Artifact Quality81.573.167.176.8

Failure Analysis

By jointly examining generated artifacts, rubric losses, and execution traces, xDailyBench identifies two core failure mechanisms. Both can involve substantial agent activity, yet they break down at different stages of producing a reliable result.

01 Feasibility judgment

Detailed Planning without a Feasibility Check

The agent moves directly into detailed planning when constraints on space, budget, schedule, work limits, and risk are scattered across the prompt and attachments, but never verifies that all conditions can hold simultaneously.

Scattered constraints Detailed plan produced No joint feasibility check
Paper case · Older Home Renovation Plan
  • The proposed 14-week schedule exceeds the three-month move-in deadline.
  • Only RMB 10,000 remains for contingencies, while a low attic is assigned as a treatment room.
  • The claimed on-time, on-budget delivery is therefore unsupported.
02 Stopping judgment

Evidence-Collection Non-convergence

Because open-ended information tasks have no natural retrieval endpoint, the agent mistakes successful search calls for progress and keeps adding queries instead of deciding that the evidence is sufficient and moving to synthesis.

Relevant source found More searches added No synthesis or deliverable
Paper case · Data-breach Legal Analysis
  • The official source ranks first in the initial search, but the agent never opens it.
  • Query growth adds volume without new evidence; the last 60 searches yield only nine unique top-three URLs.
119searches
0pages opened
0files written
Key distinction The first mechanism converges to a polished but infeasible artifact; the second never transitions from evidence gathering to synthesis and artifact production.

BibTeX