From weight conversion, model computation to training cycles and reasoning services, T2J-Bench tested the migration quality of PyTorch to JAX with 63 set-up missions. The strongest baseline score among the 20 groups in the current agent-type model configuration is 0.676; keeping Claude-Code unchanged from opus-5, TwinHarness will increase the score from 0.649 to 0.723, a relative improvement of 11.4%.
Figure 1: The title of the paper's first page, authors and institutional information.
T2J-Bench: Benchmarking Coding Agents on Execution-Grounded PyTorch-to-JAX Repository Migration
Paper title: T2J-Bench: Performance evaluation of coding agents on an executable PyTorch to JAX migration library
The paper:
The main page of the project:https://ft2023.github.io/T2j-web/
Open source code:https://github.com/google-research/t2j-bench/
Moving a PyTorch model warehouse to JAX requires synchronizing model definitions, load loads, training cycles and reasoning caches. A seemingly small modification may change the gradient direction or follow-up training trajectory.
A team from Google Cloud AI Research, the University of Illinois at Urbana-Champaign, the University of Southern California, the University of Notre Dame, and Google DeepMind developed the T2J-Bench and designed TwinHarness, allowing agents to leverage operational raw warehouses, routine positioning, repair, and verification behavior differences.
What behaviours should be maintained when relocating?
PyTorch is often organized in a way that executes instantaneously, with state modules and object-oriented computations; while JAX emphasizes explicit parameters, functional expressions, and compilable execution. During the migration process, the agent needs to reorganize the execution of the program so that the model, trained state, and running input work together in a new framework.
The paper gives an intuitive example: first the average for the error squares, then the average for the average squares, and only one calculation sequence, but different losses and gradients. The code may be running normally, but the training process has deviated from the original implementation. Similar problems can also occur in optimizer updates, attention masks, and KV caches in the reasoning phase.
Figure 2: Three migrations from the same source warehouse: retention behavior, deviation behavior, and code and loss curves still dependent on PyTorch.
There is a more concealed scenario: the candidate warehouse provides a JAX interface on the surface and the actual computation is still calling PyTorch. Its output may be very close to the source implementation, but it has not completed an independent framework migration. Thus, T2J-Bench simultaneously checks two things: whether the candidate implementation is dependent on the source framework, and whether it retains the evaluated execution behavior under the same input.
This also explains why the study included assessment of training trajectories and service status. Single forward-to-output is only part of system behavior, and real engineering deliveries are subject to reverse propagation, continuous updates, and multiple-request reasoning checks.
63 missions covering training and reasoning services
The T2J-Bench contains 63 self-contained warehouse-level migration tasks covering a total of 18 model families. Each task provides PyTorch source trees and open weights from a real code library, the source trees typically contain about 15 to 20 modules. Agents are required to deliver native JAX/Flax warehouses, maintain required file structures and public interfaces, and achieve checkpoint conversions that do not rely on PyTorch.
The training section consists of a combination of 15 model families with three target categories SFT, DPO, PPO, and 45 LoRA mitigation tasks; the service section consists of six model families with a single sequence, incremental batch processing, pre-filled segmentation of three paged-attention patterns, and a total of 18 tasks.
Figure 3 T2J-Bench's mission is to build, migrate delivery and independent evaluation processes.
The evaluation is divided into three dimensions: Spec inspection structure, interface and framework independence; numeric comparison values such as module output, loss and gradient; Behavioral inspection training dynamics and behavior and state in the reasoning process. Candidate code is executed in an isolated, offline and PyTorch-free environment, and compared to reference results recorded in advance from the original PyTorch.
The final reward is a sequence of scores between 0 and 1: the specification check is a combination of powers such as multiplication gate control, numerical values and two default behaviours. Therefore, 0.723 indicates a score that cannot be directly explained as a 72.3% success rate. The reference JAX implementation itself also accepts the same set of scores, whose 0.980 scores are used to verify the task's resolvability; the true value is still from the original PyTorch implementation.
TwinHarness transforms the source code into executable specification
Cross-frame migration has a naturally available source of feedback: the original warehouse itself is able to run. TwinHarness explains the executable specifications around this plug-in.
During the diagnostic phase, the current code is read with evidence that is already running, the migration gaps are located and the repair targets are identified; during the migration phase, the JAX/Flax implementation, weight conversion and training or service paths are modified; during the verification phase, the updated warehouses are executed, the test results are recorded, and the evidence is delivered to the next round of diagnosis.
Figure 4 - The diagnosis, migration, verification cycle of TwinHarness, as well as the source-based differential checks and candidate side checks.
Verification consists of two types of signals. One type puts PyTorch and JAX into matched inputs, contrasting output structures, fields, and observable behavior; the other directly checks candidate implementations, such as whether modules can be imported, whether a parameter tree is reasonable, whether gradients are valid, whether JAX tracing is passed, and whether training inputs can be executed.
Each conclusion is labeled as either confirmed or unconfirmed. The test tool does not give a conclusion, and the candidate code is confirmed to be wrong, in two different states. TwinHarness uses only the resources available to the agent for the task, without reading hidden tests, freezing ratings references, evaluator codes or benchmark rewards, and developing feedback and maintaining separation from the final score.
The migration score under the same model and agent improved by 11.4%.
The study evaluated five agent harnesses and their supported models, with a total of 20 groups already configured. Each experimental set run three random seeds over a full 63 missions, reporting mean values and standard differences between seeds; candidate warehouses are scored by the same independent assessment process.
The best-performing configuration OpenCode + opus-5 has an overall score of 0.676. Its Spec score has reached 0.927, but the Numeric and Behavioral scores are 0.670 and 0.741, respectively, indicating that there is still a significant gap between structural specification checks and behavior assurance.
Figure 5 shows the main result table of the paper. R represents the total migration score, the subset of which is the standard deviation of the sample of three random seeds; the rest of the columns show the breakdowns and the results of training and reasoning. The complete base line of 20 also includes the OpenHands and Terminus-2 configurations in the appendix.
On a fixed basis of Claude-Code + opus-5, TwinHarness increased the overall score from 0.649 to 0.723, an absolute gain of 0.074, a relative increase of 11.4%; the strongest current configuration was 0.676, an increase of about 7.0%.
The segmental results showed that the numerical score was raised from 0.618 to 0.730, and the behavior score was raised from 0.728 to 0.792. The improvement did not cover every segmental task: for example, the reasoning total for OpenCode + sonnet-5 was still higher than TwinHarness, which was also farther away from the reference implementation of 0.980.
What are the cost options for higher quality migration?
To judge the practicality of the new process, it is also necessary to put the quality into the cost coordinates. The paper compares the API costs of completing 63 tasks per round, and adds normal run times to Claude-Code and OpenCode to see how much additional modifications can bring in.
Figure 6 - Weighting of migration quality and API costs. The horizontal axis is a cross-sectional, vertical axis is a cross-sectional axis; x1 represents the initial run, x2 and x3 represents an additional round and two rounds of normal running, respectively.
At the cost of a graphical API that is close to TwinHarness, the additional two rounds of normal running Claude-Code and OpenCode are approximately 0.66 and 0.60, respectively, while TwinHarness is 0.723. This indicates that structured verification feedback supports higher-quality migration in these configurations and budgets.
At the same time, the initial OpenCode was achieved at a lower cost of 0.676. It and TwinHarness were both at the forefront of the observed Pareto: the initial OpenCode offered a lower-cost option, while TwinHarness offered a higher reliability option. The result was a trade-off between quality and cost, which cannot be interpreted as TwinHarness being the cheapest under all budgets.
The cost curve also reveals a remarkable phenomenon. OpenCode dropped from 0.676 to 0.628 after an additional round and about 0.60 after two rounds. More changes may disrupt the behavior that is already correct, and the quality of the feedback directly affects the value of further iterations.
Upgraded four-group harness and backbone configuration
The study further compared the performance of four groups of matched harness backbone configurations, each with fixed underlying agents and models, before and after joining the process.
Figure 7. Total reward of four sets of matching configurations -- pink based agent, blue TwinHarness; each set uses the same underlying harness and model -- original paper Figure 5.
On Claude-Code, the opus-5 score increased from 0.649 to 0.723, and on Sonnet-5 from 0.408 to 0.467. On Gemini-CLI, the gemini-3.7-flash increased from 0.102 to 0.222, and on Gemini-3.8-flash from 0.087 to 0.217.
These results support TwinHarness' applicability to different harnesses and backbones that have already been tested. For existing agent systems, the research suggests an improvement pathway that can be attempted: allowing more systematic access to source execution, differential inspection and diagnostic evidence to the migration process, while retaining models and underlying execution tools.
The scope of this group of experiments was matching configuration comparisons on the T2J-Bench, which have not yet introduced generalization capabilities for unknown task distributions. Subsequently, this process still needs to be tested in different frameworks, warehouse types and deployment conditions.
How does structured feedback affect subsequent repairs?
The study also compared the changes from round one to round two. For fixed opus-5, the regular OpenCode run decreased from 0.676 to 0.628, while TwinHarness improved on OpenCode from its first round result of 0.632 to 0.706. The latter did not lead in round one, with advantages in subsequent diagnosis and correction.
Results at the task level provide another way of looking at it: there are 22 normal run-off missions in the second round of scoring relegations, with TwinHarness having two. The paper will compare the changes above 0.01 as a rise or fall, and the absolute changes below 0.01 as a steady; the two methods are compared with their own first round candidates respectively.
Figure 8 OpenCode + opus-5 is the second change in 63 missions. The normal run is 7 ups, 34 downs, 22 downs; TwinHarness is 11, 50, 2.
These phenomena are consistent with structured verification, which helps to maintain an explanation of the correct behaviour, but without individually identifying the causal contribution of a particular component, and does not mean that continuing to increase the turnaround will necessarily continue to improve.
T2J-Bench dismantled the warehouse migration into executable, comparable behavior checks; TwinHarness showed how to get the source code available in the middle of these tasks involved in the development of feedback. With the benchmark code open, researchers can compare their own agents in a unified task and evaluation process, further answering which execution evidence best helps the system retain cross-framework behavior.
Information from the author
The paper was co-authored by Tao Feng, Jiefeng Chen, Linxin Song, Yufan Zhuang, Yue Huang, Bhavana Dalvi Mishra, Chi Wang, Xiangru Tang, Jieyu Zhao, Ge Liu, Thomas Pfister and Jinsung Yoon.
The work was completed while Tao Feng and Linxin Song were research students at Google Cloud AI Research, with collaborative institutions including Google Cloud AI Research, UIUC, USC, Our Lady University and Google DeepMind.
© THE