Running the Vision-Language-Action (VLA) model on RTX 4090 workstations is no longer a particularly difficult task.
But the real challenge is: how do you get VLA models with hundreds of millions or even billions of parameters to leave the workstation, run into the robot's body, and stay fast enough to respond?
For mobile robots, relying on external high-end GPUs is not ideal. On the one hand, high-power computing devices will significantly compress battery life, shorten the robot's working time, and limit the application of the robot in a laboratory demo; on the other hand, communication between the robot and the computing device will also introduce additional delays and limit the range of activity of the robot.
But if you deploy a VLA model such as π0π0.5 directly to an on-board computing device (such as the NVIDIA Jetson Orin), the problem is equally obvious.π0.5 takes about 1.4 seconds to complete a reasoning on the Jetson Orin, and the corresponding control frequency is only about 0.7 Hz.
Figure 1 Compares the power consumption of different computing devices, robot duration and VLA control frequency. High-end GPUs can provide lower reasoning latency, but significantly increase power consumption; side devices such as Jetson Orin use lower power, but face the problem of insufficient control frequency.
To address this issue, a team of researchers from Peking University, AIRS and the PrimeBot Research Institute has proposed Jetson-PI: a real-time VLA control system for low-power end-to-end devices.
Jetson-PI did not simply continue to compress the model, but instead started at the same time from the three levels of the asymmetric reasoning algorithm, model scheduling and the underlying reasoning engine. On the NVIDIA Jetson Orin, it will increase the control frequency from 0.70 Hz to 6.06 Hz, up 8.66 times from the native PyTorch reasoning, without losing model accuracy.
This breakthrough may mean that physical intelligence is undergoing a key shift from laboratory demo to industrial applications: Jetson-PI extends the effective working time of robots through sustained, real-time and efficient end-to-end reasoning, allowing VLAs to move from being "demonstrated" to "long-term".
The paper was titled: Jetson-PI: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inference.
Address of the paper:https://arxiv.org/abs/2607.12659
This is the first time I've read this post.https://github.com/PKU-SEC-Lab/Jetson-PI
This is the first time I've seen this.https://github.com/PKU-SEC-Lab/Jetson-PI-Edge
The problem with VLA on the edge isn't just "thinking too slowly".
In the traditional synchronous reasoning mode, the robot must wait for the model to generate a complete block of action before starting to execute. The computing power and memory bandwidth of the edge device are limited, and the reasoning delay is directly expressed as the robot's long pause. A natural solution is asynchronous reasoning: the robot executes the current block of action while the model predicts the next block of action in parallel.
This, while concealing some of the time of reasoning, introduces two new problems.
The first problem is the perception and execution of errors. The model predicts the next block of action using images collected at the beginning of the reasoning. But when the reasoning ends and the action actually begins to be executed, robots and the external environment have already changed. The longer the delay in reasoning, the greater the deviation between the model "environment seen" and the "environment occurring".
The second problem is that the response time is too long. Even if reasoning and action execution are achieved in parallel, the robot's response speed to environmental changes is still limited by the complete delay of VLA reasoning. On terminal devices such as Jetson Orin, one reasoning may exceed 1 second, making it difficult for the robot to respond in time to objects sliding or changing position in complex operations.
Past asymmetrical reasoning attempts to merge new and old trajectories in the time dimension, thus improving the smoothness of the trajectory. However, its motion prediction process still has the above two problems. When deployed at the edge of a VLA model performing complex tasks such as overlapping clothing, it may result in mission failure due to movement errors.
Jetson-PI therefore chose a different route: not to merge new and old trajectories, but to directly predict future environmental characteristics and start predicting action from the future moment.
Figure 2 Synchronous reasoning, as opposed to the Jetson-PI method of execution. Although it eliminates the pause between the blocks of motion, it produces a predictive loop execution error; Jetson-PI mitigates this problem with future corrections.
Not predicting future images, but modifying future expressions
Jetson-PI's core method is known as Foresight-Aligned Asynchronous Correction, meaning "forward alignment asynchronous correction".
The team trained a lightweight future modification module. The module receives two types of input: the environment representation generated by the VLM at the current time, and the sequence of movements that have already been submitted to the robot. On this basis, it predicts the environment representation of the VLM environment at the future time when these movements are executed.
Here's a key design: Jetson-PI did not attempt to generate a complete future image, nor did all of the KV Cache for layer-by-layer modification of VLM. The former was too expensive to calculate, and the latter also introduced a lot of additional reasoning. Jetson-PI only compressed and predicted the last layer of the VLM environment, then sent the modification information to the action experts.
In order for the module to adapt to different hardware and different reasoning delays at the same time, the training process randomly samples future time frames. Whether the model is deployed on Jetson Orin, Jetson Thor, or on higher-performance GPUs, the same set of modification modules can adjust the predicted time range based on actual delays.
Figure 3 Predictive alignment asymmetrical modification framework. Future modification modules are generated by the action experts based on the current compression environment representation and the action that has been submitted, predicting the future environment representation when the reasoning is completed.
Predictive movements don't need to run the entire model every time
After the introduction of future modifications, theoretically, after completing a VLM environmental observation, the system can only repeatedly call on future modifications modules and action experts to continue to generate follow-up movements. This can avoid repeating time-consuming visual language reasoning every time a movement is generated, thus increasing the control frequency of the robot. However, due to the existence of errors in future modifications modules, accumulation of errors can lead to errors in the prediction of the movement if it depends only on future modifications over a long period of time.
The team re-analyzed the division of VLM and motion experts into robotic controls: VLM's main role is to re-observe the current environment and correct the system's understanding of the outside world; motion experts are responsible for generating high-frequency motion.
Jetson-PI therefore proposed a confidence-based scheduling mechanism. While future modification modules predict future environmental characteristics, they also output predictive characteristics. When the confidence given by future modification modules is higher than the threshold value, the system skips VLM and directly uses modified environmental information to call the action experts. Because the action experts reason faster, robots can generate new movements more frequently. When confidence drops to below the threshold value, the system re-calls VLM, reads the latest camera images, and refreshes the environment characteristics, the KV Cache and the required state caches for subsequent predictions.
Actual test cases show that confidence is usually high when the movement is steady, and environmental changes are easily predictable; while confidence drops significantly during key steps such as grabbing, touching, and placing, triggering a re-observation of the environment by the VLM.
This is equivalent to letting robots learn how to judge: "When to continue to act with consciousness, when to look again".
Figure 4 Jetson-PI confidence scheduling mechanism. When future prediction confidence is high, the system skips VLM to directly call action experts; confidence drops and trigger VLM updates during the more difficult stages of environmental changes such as crawling and placement.
Redesigned an end-side VLA reasoning engine based on llama.cpp
The algorithm solves the problem of "when to call the model", but in order for π0、π0.5 to truly meet the end-side real-time control requirements, the underlying reasoning system also needs optimization. The research team based on llama.cpp developed Jetson-PI-Edge and made three core optimizations for the execution characteristics of VLAs.
VLA reasoning is different: when the number of cameras, image resolution, and action dimensions are fixed, most of the size of the input voltage is also fixed. Although the language instructions vary in length, but are usually shorter, they can be unified to the fixed length by padding.
2) Permanent intermediate caches. Intermediate results such as visual embedding, KV Cache, etc. need to be transmitted between visual encoders, language models, and motion experts. The General Reasoning Framework writes these results back to CPU memory and then replicates them to the GPU by the next module. For end-to-end devices with limited memory bandwidth, this kind of back and forth transmission costs significantly. Jetson-PI-Edge reserves a GPU buffer for fixed-sized intermediate results, allowing ViT, LLM and motion experts to directly copy data on the GPU.
3) Flow matching develops. Pi-series models require multi-cycle flow matching to eliminate noise. Repeated scheduling and kernel launch costs are generated if each step is called separately. Jetson-PI-Edge will extend multi-cycle noise to a unified computational chart. Although the cost of first building a chart increases, it can be repeated afterwards, making it more suitable for long-running control cycles for robots.
After fully enabling GPU-intermediate caching and flow matching, π0.5 decreased the single reasoning time on Jetson Orin to 412.9 ms, with action experts decreasing the reasoning time from 536.8 ms to 123.1 ms. Combined with confidence scheduling, the system response time further decreased to 165.1 ms, with a control frequency of 6.06 Hz, up 8.66 times from the native PyTorch framework.
It is worth noting that system acceleration is not a prerequisite for quantization or cutting. The computational graph replication, caching and optimization in the paper does not change the model calculation itself, and can theoretically be combined with methods such as quantization, cutting, to further reduce delays.
Figure 5 The delay, response time and control frequency of different optimization phases on Jetson Orin.
The longer the delay, the more important the future environmental correction
In the LIBERO Spatial, Object, Goal and LIBERO-10 subset, the team compared π0.5 to Jetson-PI, RTC, and VLASH. Under different asymmetrical delay settings, the average success rate for Jetson-PI was 14.8 percentage points higher than for VLASH and 3.9 percentage points higher than for RTC.
As delays increase, the performance of VLASH, which predicts only future robot states, drops significantly. Jetson-PI maintains a more stable success rate for longer delays by predicting future environmental characteristics. For example, with the delay step Δ = 9, the average success rate for all four datasets is 45.6 percentage points higher than VLASH and 7.0 percentage points higher than RTC.
This group of experiments showed that the problem with the edge-side VLA is not just "to make reasoning fast". When reasoning delays cannot be completely eliminated, it is also necessary to make the model understand how the robot's actions during this time of delay will change the future environment.
Figure 6 Success rates for Jetson-PI, RTC and VLASH in four LIBERO sub-data sets and different asymmetrical delays. As delays increase, the performance of Jetson-PI becomes more stable.
Real robots are folding clothes
The team also conducted real-world experiments on the X2-W robot at the PrimeBot Research Institute.
The robot is equipped with a head camera and two wrist cameras, with an image resolution of 224 × 224. The team will break up the complex tasks of picking up, unfolding and folding clothes, organizing and storing them into three subtasks and performing the movements at a frequency of 15 Hz.
As shown in the figure, synchronous reasoning needs to wait for the entire action block to be generated and then continue to execute, so the robot will notice a significant pause between the action blocks and overall execution slows down. Simple asynchronous reasoning Although prediction and execution are combined to reduce the pause, it does not resolve the deviation between the moment of perception and the moment of actual execution of the action, resulting in poor continuity of the action. Real-time chunking further improves the welding between the action blocks, but still generates movements based on delayed environmental observation.
In contrast, Jetson-PI predicts future environmental characteristics based on submitted actions to align the newly generated actions with actual execution times. The robot motion trajectory in the diagram is more continuous, the clothing operation process more smooth, while maintaining higher real-time.
Figure 7 Jetson-PI performs clothing folding tasks on X2-W robots.
Location of Jetson-PI
Jetson-PI's focus is not on continuing to push the parameters of the VLA model higher, but on solving a problem that is closer to a real robot landing: how to keep it fast enough when the model has to run on an on-board device with limited power, bandwidth and heat dissipation?
The answer to this work consists of three parts: to mitigate asynchronous reasoning by using future environmental representations; to reduce unnecessary VLM calls with confidence scheduling; and to execute expense on the compression-side with a system-optimized approach to VLA characteristics. For mobile robots that emphasize power, durability, range of activity, and offline operation capabilities, this approach provides a more viable route: without having to move the entire high-end workstation, it also allows VLA controls to be gradually closer to real-time.
Introduction by the author
The first author of the paper is Ph.D. student in the second year of the Beijing University Institute of Artificial Intelligence and a mentor to Prof. Li Chen. His main research focuses on efficient AI algorithms and systems, efficient reasoning and edge computing of big language models, focusing on algorithm simulation system simulation hardware collaboration design, and focusing on solving the energy efficiency and real-time bottlenecks of AI systems in real deployment.
He has published several papers at AI and computer architecture summits such as NeurIPS, DAC, ICCAD, and related works include: efficient end-to-end target navigation algorithm EfficientNav, body-planning memory management system KEEP, layer-leaping efficient logic framework DySL-VLA for robotic operations, a lightweight language model deployment system MCUBERT for edge information processing, and an asymmetric logic framework Jetson-PI for end-to-end real-time robot control, among others.