It's the first time in the history of the world.
Agents are now skilled at calling tools, modifying files, running tests, but in the face of a large software system that has evolved over a long period of time, can they start with a new demand and deliver full functionality?
Product managers come up with new requirements, and developers often don't get document lists or function signatures. First, they need to understand constraints, track module dependencies, and then decide where to change, how to change, how to prove a change. These analyses affect subsequent delivery.
LoLBench was proposed by organizations such as Ren Tan University, City University of Hong Kong, and Chinese University of Hong Kong, which put Agent into a software system with an average of about 2.4 million lines of source code, evaluating the entire process from feature proposals to module delivery, and testing.Putting the need into the perceptual capacity of the system and the ability to implement cross-module modifications。
Link to the paper:https://arxiv.org/abs/2609.37143
The data collection:https://huggingface.co/datasets/lolbench26/LoLBench
The experiments revealed key disruptions: Agents may change locations, but omit synchronous changes that must be made in other modules. Most tasks remain unresolved even if the instructions are added.There is still a long way to go from understanding needs to completing deliveries.。
Redefining long-range code tasks
What kind of long-range code task has real engineering value?
The "length" cannot only look at the time taken, the number of rounds or the consumption of tokens. For example, allowing an agent to output a number every minute, from 1 to 1000, while taking a long time, requires a simple loop. The SWE-bench type local bug repair task cannot cover the challenges from demand analysis to module delivery.
LoLBench's focus is on long-range code tasks.Modular development for complex systemsAfter receiving new requirements, developers repeatedly analyze: which behaviors must be implemented, whether existing architectures are supported, which dependencies will be affected, how to verify.
These tasks also haveUnderstanding the ComplexityandReaching ComplexityThe first is reading proposals, identifying implicit constraints and cross-module dependencies, deriving the difficulty of modifying the scope and implementation of the specification; the second is converting the specification into the correct cross-file modification and avoiding the difficulty of damaging existing functions through testing and repairs.
Figure 1 compares perceived complexity with realised complexity.
How LoLBench evaluates both types of complexity simultaneously
From functional proposals to implementation
LoLBench selected 100 tasks from the Community Features Proposal ("Enhancement Proposal") and the corresponding merged PR, covering 29 systems, 8 languages and 5 fields.
The Agency only receives proposals with the implementation of pre-code repositories, proposals averaging about 5,000 English words. Evaluations are conducted in a disconnected Docker environment to prevent Agent retrieval upstream implementation.
For example, PEP 617[1] proposes to replace CPython's LL(1) analyzer with PEG, which requires the implementation of a syntax definition, analyzer generation, abstract syntax tree and error reporting. Agents must identify the entire implementation link.
How to perceive complexityLoLBench references target class names with an average referral rate of only 7.3% in the proposal, less than half of the SWE-bench Verified (15.9%) and less than one-fifth of the FeatureBench (38.7%).
Implementing regulations for engineering deliveriesHow to achieve complexityThe LoLBench warehouse averaged about 2.4 million lines of source code, about 9.4 times that of SWE-bench Verified; 4.9 times that of FeatureBench; achieved an average modification of 5,499 lines, the latter two being lines 38 and 2050, respectively. Each task modified an average of 64.9 files, 40.1 categories and 195.9 functions; the number of files was 25 times and 4.2 times respectively.Cross-module collaboration is more difficult to verify.
Table 1 LoLBench comparisons to existing code task evaluation sets.
LoLBench delivers accuracy through two types of test verification: F2P, which requires pre-failure, post-failure, and F2P.F2P allows for different internal implementations through open interface testing behavior.The return test (P2P) requirements passed both before and afterwards.
Based on the original testing of the project, the research team constructed missing functionality, modified requirements and other errors as variants, supplemented test use cases to identify leak detection errors, and improved test coverage for reference patches.
After enhancement, detection rates for 1,688 variants increased from about 61% to 96%, and F2P line coverage by mission average increased from about 60% to 79%.
28 combinations of Agents and bottles
Changing models or scaffolding, Agent's highest resolution rate is only 14%.
The team evaluated 28 available combinations of six front-end models and five armrests, with a single-mission time limit of six hours. Opus 5 with Claude Code or a mini-SWE-agent combined with a maximum resolution of 14%, with an average of only 2% in all combinations.
The main results of experiments in Table 2 paper
The Opus 5 mini-SWE-agent has the same resolution rate as Claude Code, which takes 3.2 times as much time as the latter, generates 5 times more tokens, and consumes more uninterrupted and higher resolution rates.
A lot of interaction with Tokens is spent on code positioning.In terms of token consumption, the positioning phase, on average, represents the non-cache input of the token.52.4%Generated by Token45.4%In terms of the number of Agent interactions, the number of interactions in the positioning phase is also significantly higher than in other phases.
Figure 2 Average number of interaction rounds at each stage.
Controlling the modification of existing files and functions, all 28 groups of Agents have a document modification accuracy of 38.0% to 80.4%, recall rate of 30.3% to 65.0%; function modification accuracy of 54.2% to 65.6%, recall rate of 20.8% to 53.4% code.Low positioning accuracy and incomplete coverage.
After adding the revised guidelines, Agent is still struggling to deliver in fullThe research team retained the original functional proposals and provided additional references to implement the exported modified file trees and API implementation specifications, helping Agent determine the scope of the modification. The resolution rate for Opus 5+Claude Code increased from 14% to 34%, GPT-5.6 Sol+Code increased from 3% to 25%, and DeepSeek V4 Flash+OpenCode increased from 1% to 17%.
Table 3 Supplements the experimental results of modifying the file tree with API implementation
Additional modification guidelines can alleviate the perceptual burden, but the best combination remains.66% of missions unresolvedAgents still need to coordinate modules, implement interfaces and pass verification, which suggests that the complexity of implementing large systems is just as high.
From code generation to large software systems
These results provide a verifiable direction for Agent R&D: whether warehouse retrieval covers critical dependencies, whether pre-coding implementation specifications are complete, and whether verification can detect cross-module omissions in a timely manner.
The value of software engineering lies in whether user needs can be reliably met. Agents need to understand why changes occur, how they affect the system, and to implement the changes as a whole. LoLBench wants to drive the continued evolution of Agent capability assessment from generating code to supporting software.Make delivery results a measure of Agent's ability to test。