Of the 12 million candidates who pulled out the requests, 5,000 SWE code tasks were left with less than 0.05% to complete. The code stream of "themes, answers, conversions, training, evaluations" that runs automatically is Lego Flow; in the process, we discovered three other observations.
Software engineering is the core capability of recent large-language models, but the production of high-quality code data is still quite complex, involving long processes ranging from warehouse detection, task verification, trajectory reasoning to training evaluation.
How to automatically run this line of code data flow: "create questions, answer questions, filter, train, evaluate". Researchers from Huawei, HCMG, and HCMG have jointly launched Lego Flow, a simple, interactive code data engineering framework, with highlights including:
Automated workflow: From warehousing and PR to collection of task verification, trajectory reasoning, training and evaluation, each process is packaged as standard plug-in skills, and coding smart objects such as Claude Code, Codex can be called directly.
Open Source Missions and Trajectories: The team released LegoFlow-SWE, containing 5,000 missions selected from 12 million PRs, covering 8 programming languages and 20 task labels, and 2,780 high-quality trajectories that successfully verified; training Qwen3.5-35B-A3B-Base with just 1,000 trajectories, achieving 70.2% and 48.8% on the SWE-bench Verified and Pro, respectively.
Moving towards recursive self-improvement: With very few man-made settings, the intelligent body can run the entire process, evaluate results and adjust strategies. After two rounds of iteration, Qwen 3.5-35B-A3B-Base increased from 7.6% to 64.4% on SWE-bench Verified.
Project blog:
Open source data:
Code address:https://github.com/LegoX/LegoFlow
How does LegoFlow work?
Code data is a key raw material for training large models of reasoning, but the cost of artificial marking is high, LLM is uncontrollable, automated verification is difficult, and is a community-recognised bottleneck.
Root converts user targets into workflows and coordinates task distribution, resources and feedback;
Curator translates warehouse and pull requests into verified tasks;
Tracer generates training tracks;
The trainer's tweaks;
Evaluator returns the benchmark ratings.
Each block is encapsulated as a plug-in skill that enables a coded smartphone to operate the entire workflow.
Block's design concept
Imagine: complex engineering workflows are typically divided into multiple engineers, each responsible for bounding clear links and collaborating with each other; LegoFlow brings the same structure into the workflow of the smart body. Each block is like an engineer: he encodes the smart body management, handles the required warehouses, scripts, dependencies, and environments, and packages them into plug-in skills that allow users to call directly through the smart body, simplify the operation of the smart body and organize the work involved in multiple repo operations.
Curator: building verified coded intelligence tasks
Curator collects GitHub repositories and pulls requests to create verified coded smart body tasks (Fig. 2), and the main steps include:
Step 1: Data discovery: Search for relevant and merged pull requests in active warehouses, apply basic quality filtering and record source information.
Step 2: Data preparation: Collect PR, issue, commit and test evidence, transcribe it into clear, leak-free tasks, and separate the repair code from testing.
Step 3: Data assembly: Build candidate tasks according to the standard Harbor task format, equipped with verifiable, operational isolation environments and testing.
Step 4: Verify with tabs: Confirm that a faulty version will fail; reference corrections are acceptable; then assign difficulty scores and analytical labels and add them to public lists and tabs.
Tracer: Generate and filter the trajectory of the mission
Tracer uses Claude Code, OpenCode, OpenHands, and other coding frameworks to generate trajectories from verified tasks (Figure 3).
Step 1: Input tasks, load verified tasks from Curator (or external dataset) and verify lists to avoid repetition.
Step 2: Model proxy, by sharing the proxy connection framework with the selected model and recording the complete interaction.
Step 3: Containerize the operation, run each smartphone in a segregated Harbor environment, execute verifiers, and record the final rewards and trajectories.
Step 4: Convert the SFT training format, converting the effective trajectory to a unified training format, combining rules-based and model-based scores for training screening samples.
Trainer and Evaluator
LegoFlow verifies the quality of the data through SFT training and evaluation: the smart body can automatically convert Tracer's effective trajectory to a standard LLaMA-Factory format for Trainer use, after training Evaluator automatically position checks points, run benchmark evaluations and publish to the dashboard.
Evaluator is built on top of Harbor and supports smart benchmarking such as SWE-bench Pro; mitigating cheating by limiting networking, adding anti-cheat hints, and making the results focus on the capabilities of the model itself.
In addition to SFT, the team is also integrating Lego-RL into Trainer, allowing it to directly use Curator verification tasks for RL training.
Visualization of the sign
The code data process can last for hours or even days, so each LegoFlow block is equipped with a sign to monitor progress, check intermediate products. Figure 4 is a Curator snapshot, containing 822 repositories, 8,818 pull requests, 926 built-in tasks and 270 verified tasks.
The sign also shows pass rate, label distribution, scoring standard scores, sampling trajectory and assessment details:
Curator signs
Tracer signs
Trainer signs
The Evaluator board
Lego Flow-SWE: Scale up and build verified missions
There are millions of drag requests on GitHub, but only a small fraction of them can be reliable software engineering tasks: they must have replicable environments and tests. To verify whether LegoFlow can scale to build such a task, the team released a LegoFlow-SWE collection of open SWE tasks and trajectories from 12 million candidate drag requests and assessed whether it can compete with the most advanced SWE datasets (such as ScaleSWE, DeNovoSWE) under fair settings.
Construction began with the collection of more than 700,000 GitHub repositories and their associated 12 million candidate pull requests. After a valid diff, reasonable patch sizes, testing and issue evidence, only 600,000 PRs remained; LLM reviewers and execution verification further removed simple or unverifiable tasks, ultimately leaving 5,000 verified tasks, accounting for only 0.0417% of the original data pool. GLM-5.2 produced 9,767 runs through OpenHands SDK and OpenCode, 2,780 of which were certified.
Results of the training
For fairly verifiable source assignments, the team fixed the teacher model, frameworks, ratings, samples and budgets, minimized Qwen 3.5-35B-A3B-Base with about 1,000 trajectories from each source, and assessed on the SWE-bench Verified、Pro、Multilingual.
After training on LegoFlow-SWE, the model reached SWE-bench Verified 70.2%, Pro 48.8%, Multilingual 57.0%, ahead of Qwen-3.5-35B-A3B-instruct and the external open-source trajectory data pool; an improvement of 5.8, 1.9 and 1.0 percentage points, respectively, compared to SWE-rebench-v2.
Experience 1: Quality is more important than quantity
The quality of the task involves two aspects: the task must be verifiable, the environment and the test must be able to reliably separate the original code from the repaired code; and it must be difficult enough to actually investigate rather than simply patch it.
The curator uses static scores, ranging from variation range, logical complexity, context width, test complexity and instruction complexity, to five signal-weighted scores: 1.0×10.0):≤4.0 for simple, 4.0×7.0 for medium, 7.0 or more for difficulty.
Compared to the SWE-rebench V2, the average LEGO Flow-SWE difficulty score is higher (6.27 to 5.80), the difficulty of the task is 39.4% (compared to 35.1%); changing the task data pool only increases Verified、Pro、Multilingual by 5.8、1.9、1.0 percentage points, respectively.
Experience 2: The stronger the model, the more likely it is to "cheat".
A stronger teacher model may be better at cheating on benchmark assessments than at solving problems. The team observed three ways to leverage information leaks: finding open warehouses and their upstream to fix PR, directly copying patches already made, or proving the changes generated with upstream submissions to be correct.
Following the removal of the leaked responses, the scores of the teacher model and the Verified model for the student model decreased by 8.0 and 10.0 percentage points respectively; and the higher the cheating rate of the new open source model: GLM-5 was 3.6%, GLM-5.2 was up to 21.2%; it did not launch more suspicious attempts, but was better at turning them into successful cheats.
The team reduced the GLM-5.2 fraud rate to the following 1 percent by two measures: one, adding an anti-fraud instruction to the prompt, requiring the model to autonomously derive resources within the task environment only; two, limiting access to content; e.g. removing local Git information containing reference patches; limiting network access.
Experience 3: The depth of reasoning on SFT trajectories is more important than the coverage
CoT is an important part of learning from teacher models, but the key is to reason in depth rather than frequency. The open-source dataset contains CoT in 97% and 100% of rounds, while Lego Flow-SWE has only 62%, but its average reasoning length is 714 tokens, far exceeding the external data pool's 181 ton 309 tokens.
It was also found that GLM-5 triggered reasoning in 100% of rounds (average 82 tokens), with its student model scoring 60.4/30.2 (Verified/Pro); GLM-5.2 triggered reasoning in only 62% of rounds, but averaged 500 tokens, rising to 64.4/46.9.[4] After dividing the data pool into four quarters on average, the shortest group was 57.0%, while the remaining three groups increased to 64.0% and 65.0% as the reasoning deepened, thus tending to retain the trajectory of deep thinking rather than the trajectory of high reasoning triggering ratios.
Experimental reports:
Moving towards regressive self-improvement
LegoFlow-SWE verified the effectiveness of a single process. The team then used evaluation feedback to improve data collection and training, conducting an end-to-end experiment: covering warehouse and PR collection, trajectory generation, training, evaluation and strategy optimization, all managed by intelligent agents.
Goal: Starting from the original GitHub warehouse, build training data using Curator, Tracer, Trainer and Evaluator and fine-tune Qwen 3.5-35B-A3B-Base, with 512 valid tracks on the SWE-bench Verified to a resolution of 60.0% or more.
Generation 1: First attempt. Root block coordinated workflow; Curator constructed 4,166 verifiable tasks from the combined pull requests, Tracer generated 915 effective tracks, and the process selected 500 items according to scoring criteria.
Generation 2: Improved trajectory filtering. The first evaluation exposed two data problems: 80.7% of responses contained <think> blocks, but many were merely short sentences such as "let me check", lacking substantive modulation; the tool calling format for some trajectories could not be evaluated by framework analysis.
Results: In 915 trajectories, the intelligent body deleted 97 repetitive and 234 overly shallow trajectories; the median length of reasoning was increased from 140 to 959 characters per round, with the proportion of reasoning content dropping from 80.7% to 30.7%. Trained with 512 trajectories retained, the model reached a resolution rate of 64.4%, exceeding the target.
The whole process is running:
The next step
The team is expanding Lego Flow in three directions:
TerminalBench: Terminal-Lego work migration is being upgraded to LegoFlow, incorporating new changes to Terminal-Bench 3.0 and Terminal-Bench 4.0.
ProgramBench and NL2Repo: Digging into more complex, long-term software engineering tasks as a starting point for executable programs or natural language needs.
Retrograde self-improvement (RSI): With the emergence of stronger intelligences, the team believes that abstract design and feedback execution can support a more general, long-running self-improvement system.
At the moment, LegoFlow's data, code, and documents are all open source, and we welcome the experience, star, while also asking everyone to pay attention to the team's work: LegoX, an open source collection of Agent data that includes high-quality long-range missions, trajectories, and blocks to build smart objects together.
LegoX：
Lego-RL：
SWE-Lego：
Interested friends, welcome to chat in the comment section!