Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
Abstract
DEPTH generates complex terminal tasks by measuring sequential dependency depth rather than turn count, producing harder benchmarks that improve agent fine-tuning.
Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation. We find, however, that the turn count largely reflects the habits of the model, not only the demands of the task, and that the labels barely track how often models pass. We describe a terminal task by its work, the number of operations it requires, and its depth, the number of turns that must happen in sequence because the commands of each turn can only be chosen after reading the output of the previous one. Estimating depth as the smallest turn cap, stated to the agent, under which the task passes, we audit 734 tasks from six existing datasets. Synthesis has mostly added work, not depth: 94% of the solved synthetic tasks pass within three turns, yet depth, unlike the turn count, predicts difficulty beyond work. We propose DEPTH, in which a generation agent writes each task one segment at a time in a real software environment, carrying out each segment before writing it and starting the next from the state it leaves. Our 105 tasks exceed every audited dataset in depth and work, and none of six current models passes half of them. Fine-tuning Qwen3-8B on runs from these tasks improves it on two terminal benchmarks.
Community
Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation. We find, however, that the turn count largely reflects the habits of the model, not only the demands of the task, and that the labels barely track how often models pass. We describe a terminal task by its work, the number of operations it requires, and its depth, the number of turns that must happen in sequence because the commands of each turn can only be chosen after reading the output of the previous one. Estimating depth as the smallest turn cap, stated to the agent, under which the task passes, we audit 734 tasks from six existing datasets. Synthesis has mostly added work, not depth: 94% of the solved synthetic tasks pass within three turns, yet depth, unlike the turn count, predicts difficulty beyond work. We propose DEPTH, in which a generation agent writes each task one segment at a time in a real software environment, carrying out each segment before writing it and starting the next from the state it leaves. Our 105 tasks exceed every audited dataset in depth and work, and none of six current models passes half of them. Fine-tuning Qwen3-8B on runs from these tasks improves it on two terminal benchmarks.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Cross-Benchmark Transfer from RL on Agentic Coding Tasks (2026)
- LiteEvo: Automated, Cost-Efficient Harness Evolution for Generalization to Unseen Tasks (2026)
- SkillGym: Training Skill-Use Agents with Automatic Verifiable Environment Generation (2026)
- AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop? (2026)
- Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models (2026)
- Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards (2026)
- InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell? (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Hi! Great stuff!
Are you going to release the dataset as well? The links lead to 404 page.
Hi Petrovalex!
Thank you for your interest in our work!
We plan to release the dataset soon. Please stay tuned~
Hello. I was interested in the check in Appendix H of your paper, where the segments of a task are merged into one instruction. The work, the environment and the checks stay the same, yet nine of the 25 tasks are no longer solved within the original cap. You count them among the tasks whose D does not decrease and conclude that depth comes mainly from what the tasks require.
It seems to me that these nine tasks may speak less about depth than about the load of simultaneous requirements. Merging does not change how many turns must follow one another, but it does change how many requirements the model has to hold at once: according to your Table 9, the instruction of a whole task contains 83 names that the checks rely on, while the instruction of a single segment contains 14. Such a load leaves a characteristic signature. While the load is below the model's limit, quality stays high, and once the load exceeds the limit, quality begins to fall. A task that is large along only one axis therefore still fits within the limit, while a task that is large along both goes beyond it. This, it seems to me, is what your Section 4.3 shows: the tasks large in both work and depth have a mean percentile of the pass rate of 0.29, against 0.53 to 0.59 for the other groups, and a task large in only one of the two is not harder. Merging the segments adds one more load to the same work and the same depth, namely all the requirements at once, and some of the tasks go beyond the limit.
Such a limit can be described as the capacity of a model. A study of the behavior and degradation of language models under controlled load (arXiv:2610.36095) varies the load separately along several axes and shows that each axis has its own law. In the measured model, quality falls by about the same amount with each doubling of the number of simultaneous instructions, while the length of the task and the position in the response act as two interacting constraints. Capacity is derived from these laws as well: for example, instruction capacity is the number of simultaneous instructions that a model can follow at a given level of quality.
In your work these axes seem to be directly visible. Work corresponds to the length of the task, that is, to the volume of actions; depth corresponds to the position in the response, that is, to how far the solution has progressed; and merging the segments corresponds to a growing number of simultaneous instructions. The interaction of work and depth in Section 4.3 then corresponds to the interaction of length and position. If so, depth measured with all requirements given at once also depends on how the task is delivered, and it would make sense to report, alongside work and depth, how many requirements a task places on the model at once. This also concerns training: both benchmarks on which you test the fine-tuned model give all requirements in one instruction. For now this is only a conjecture: in the study mentioned above the laws were measured on simple rules that replace one word with another, and their transfer to terminal tasks has not been shown. It could be tested by merging the segments gradually, two at a time, then three, and so on up to all six: the number of simultaneous requirements would grow in stages while the depth of the task stays the same. If the failures come from the load, the share of unmet requirements should grow in a regular way with each stage, for example by the same amount with each doubling of their number, and the checks on names would show which segments' requirements the model loses first. If difficulty is set by depth alone, the stages of merging should not increase the share of unmet requirements.
Get this paper in your agent:
hf papers read 2611.01927 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper