YC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Execution

Abstract

As LLM agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduce YC-Bench, a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open-source, across 3 seeds each. Only three models consistently surpass the starting capital of $200K, with Claude Opus 4.6 achieving the highest average final funds at $1.27M, followed by GLM-5 at $1.21M with 11× lower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for 47% of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance. YC-Bench is open-source, reproducible, and configurable.

Key Findings

Only a few models build client trust; most choose clients indiscriminately

Tasks that require trust come with higher rewards and smaller workloads, yet most models maintain minimal trust (level 1–2) with all clients instead of specializing. Only 4 out of 10 models across 6 out of 30 runs explicitly maintain a whitelist of preferred clients in their scratchpad. The rest distribute tasks indiscriminately, barring themselves from the highest-return tasks.

Proportion of completed tasks requiring client trust.

Final trust level per client averaged across seeds (ADV = adversarial).

Identifying adversarial clients remains a challenge for all but a few models

Half of all models accept adversarial tasks at a rate higher than their natural market share (~32%), showing indifference or misjudgment. Two-thirds of all runs make no mention of blacklisting any adversarial client. However, the top three models accept adversarial tasks at 1/4 the rate of the next best model –they correctly spot the work quantity inflation and write explicit avoidance guidelines to their scratchpad.

Ratio of adversarial tasks among all accepted tasks. Dashed line = natural market share (~32%).

Client selection policy observed in agent scratchpads per seed.

Suboptimal employee assignment is the second-largest failure mode; cost efficiency varies dramatically

Beyond adversarial clients, 7 out of 11 models fail substantially from assigning employees whose productivity cannot meet deadlines, or from spreading employees across too many concurrent tasks. Models have perfect information about employee skills and task requirements, so these failures stem from poor estimation, not missing data. On cost efficiency, Kimi-K2.5 achieves 2.5× more revenue per API dollar than the next best model, while GLM-5 is 11× more cost-efficient than top-ranked Opus despite near-identical performance.

Failure mode breakdown: adversarial, wrong staffing, and over-split.

Cost efficiency: in-game revenue per dollar of API cost.

Four failure profiles reveal a spectrum of long-horizon incoherence

Opus rewrites its scratchpad ~34 times per run but occasionally violates its own blacklist. Flash executes a rigid 4-command loop every turn with zero adaptation, surviving through sheer throughput. Sonnet exhibits a reasoning–execution gap: it derives correct rules then immediately ignores them, averaging 7.2 concurrent tasks while its scratchpad says "one task at a time." Grok shows aware inaction: its scratchpad accurately diagnoses critical issues but it takes no corrective action, going bankrupt with just 6 days of runway after accepting a 0%-success-rate client.

Representative failure moments for four models: scratchpad state, agent action, and outcome.