The team for the paper came from the Institute of Industrial Research (AIR) of Tsinghua University, Didi, Beijing Aerospace University. The team has published dozens of papers in the field of autonomous driving and physical intelligence in recent years, and has worked extensively with renowned universities and research institutions at home and abroad.
The first author of the paper was Liu Xiaobo, a Ph.D. student at Beijing Aerospace University, who proposed the first MLLM native autonomous driving model, CuriousVLA, and was an important contributor to the current autonomous driving model, CLOVER. To date, he has published seven papers at top conferences such as ICLR, NeurIPS, CVPR, ICCV, ECCV, and has been a long-time top reviewer and recipient of the CCF Chinese Computer Society's Annual Application Award.
This article mainly introduces the latest paper from the team, Teaching Vision-Language-Action Models What to See and Where to Look, which has now been officially received by ECCV 2026, and the corresponding code and models are fully open source.
The work focuses on the training of self-reporting VLA models in autonomous driving scenarios, with the main view that existing VLA traffic-related data relies heavily on text-based visual query and mind-chain reasoning data, which emphasize language reasoning rather than action-based planning. To address this problem, the work addresses the VLA model's concerns by using the distillation of traffic elements to make it more focused on the visual elements; at the same time, the work also proposes a technique for learning the BEV trajectory into graphite space, which can make full use of the visual capabilities of the base model and make the space within the trajectory understandable for the model. DriveTeach-MSVLA was tested on NAVSIM, reaching 90.4 PDMS; further enhanced performance can be achieved by using the PDMS, which can be used to select the trajectory 92.7.
Teaching Vision-Language-Action Models What to See and Where to Look
Collaborating institutions: Beijing Aerospace University, Qinghua Institute of Industry, Didi, Chinese Media University
Link to the paper:https://arxiv.org/pdf/2607.01658
The main page of the project:https://github.com/ShivaTeam/DriveTeach-VLA
Background and challenges: Autonomous driving models cannot benefit from Internet pre-training
Imagine a not uncommon scenario: the car prepares to turn left, with pedestrians on the right approaching the zebra line, and the front car obscures part of the field of view. Thanks to the powerful pre-training capabilities learned from multimodal base models in Internet data, the VLA has now been able to well describe the response to this scenario: it should slow down, observe, gently let the pedestrian turn, and then turn left. But does this large language model really have the ability to determine the safety of the vehicle's driving control?
The paper very intuitively demonstrates the need for this mechanism through several examples, the following (a) is a common VLA model baseline that uses Qwen 2.5 - VL to adapt to self-driving data, the focus of the resulting model is very distributed on the graph and there is no clear focus focus. The problem with the paper's analysis of this focuslessness is mainly because current VLA training is actually using large language models as a large adapter, VLA is just an independent learning of how to output thought chains and corresponding trajectories according to the scene, and there is no real understanding of the relationship between thought chains and trajectory predictions.
DriveTeach-VLA's team believes that while VQA and thought chains can teach models to understand the meaning of traffic, they cannot automatically ensure that language reasoning and the corresponding planning space of control are consistent. Therefore, the idea of the paper is to allow the model to learn how to focus on the visual elements involved in language reasoning during trajectory reasoning, which ultimately affects the generation of vehicle trajectories. Simply put, it is to teach the model where to look and then teach it where to look next, thus guiding the model to focus its attention on the closely related locations of traffic lanes, pedestrians, red green lights and traffic rules.
Specifically, DriveTeach-VLA does not continue to stack a longer chain of thought to explain the text, but instead breaks up visual-language-action training into three more specific things: first teaching the model what to see, then telling it where to look in the future, and finally correcting how to drive.
What to See, DVD Pretraining)
The VLA's use of multimodal models as a foundation is due to the large number of images seen during pre-training that can provide a significant amount of world-leading knowledge. However, in specific autonomous driving scenarios, the importance of road signs and pedestrians crossing is completely different for the next second's trajectory.
In training, GroundingDINO first identified key objects such as vehicles, pedestrians, roadblocks, and traffic lights; the paper hoped that the VLA would be able to pay as much attention to these key objects as possible, so the paper referenced the work of the Oxford University Visual Geometry (VGG) on CLIP's visual prompt research that could directly change the model's attention as long as objects with clear visual identification markings on the image were used.
Teachers of ViT will naturally focus on the traffic elements marked by the bbox box, whereas students of ViT will directly learn the traits that have been corrected by the attention. In this way, the instructor's fluid traffic objects can be transferred to the student through the distillation algorithm. In terms of distillation, the paper uses a block-level sensing distillation similar to that of the Swin Transformer, that is, the characteristics are then averaged to be polarized after the pre-division of the blocks, so as to obtain as much as possible of the characteristic changes that are being brought about by the frame.
The paper also further attempted to test whether the performance gains from this pre-labeled version of GroundingDINO were stable by randomly discarding the markup box for GroundingDINO's output and disrupting the Jitter bbox box. The results showed that DVDs can deliver about 1.4 PDMS improvements, and that DVDs can still deliver good performance gains after a 20% increase in noise.
Lesson 2: Where to Look, 2D-TGP image trajectory: Where to Look, 2D-TGP image trajectory
The paper also gives an insight from the perspective of pre-baseline training, which suggests that the multi-modal model base is now imagined as a model that needs to be learned from scratch in the field of autonomous driving. However, in reality, the multi-modal model base benefits from pre-training with a large amount of data, and the adaptation of its ability to understand scenarios such as traffic and road conditions is easy to learn. The real bottleneck lies in the fact that autonomous driving planning typically outputs trajectories in the Bird-Eye-View, BEV coordinate system.
For this purpose, DriveTeach-VLA introduces the 2D Trajectory-Guided Prompt: 2D-TGP: uses camera inputs and outputs to project the expert driving trajectory from the BEV coordinate system onto the image plane, obtaining a series of pixel coordinates; then converting these pixel coordinates into text, as a prompt for input to the VLA model. Thus, the model sees no longer just a left-to-left turn, but also a more specific visual reference: where the future trajectory should roughly be located in the image.
This semantic instruction, for example, can be translated by 2D-TGP into a series of 2D image indicators that can be directly understood by the larger model.
In system design, DriveTeach-VLA uses two Qwen 2.5-VL-3B models to work separately. The first model is the TGP-Prompter, which is responsible for predicting 2D-TGP on the image plane based on current images, vehicle status and driving instructions; the second model is the TGP-Planner, which combines images, vehicle status, driving instructions and this 2D-TGP prompt to generate the final BEV trajectory.
The training method is also noteworthy. Inspired by the teacher-forcing paradigm that predicts a token shell next to the base model, the paper uses the real-valued 2D-TGP when training Planner, i.e. standard cues obtained from expert trajectory projections; but at the reasoning stage, Planner gets the 2D-TGP that the TGP-Prompter itself predicts.
However, this poses a natural problem: the model sees relatively accurate trajectory hints when training, while the test relies on hints from another model to predict, and there may be a teacher-forcing training gap between the two. The authors specially verified that the overall image trajectory can be predicted more accurately, while the quality of the image trajectory demonstrates the importance of image trajectory as a good intermediate representation. Although the final BEV trajectory prediction plays a critical role, overall performance does not collapse due to slight errors in the image trajectory.
The results of the experiments: melting, NAVSIM, nuScene
The paper provides a very detailed dilution verification of DVDs and 2D-TGPs. Qwen 2.5-VL-3B baseline is 84.8 PDMS; adding VQA and CoT increases to 86.4, indicating that language supervision is indeed useful. Continuing to add DVDs is 87.3, adding 2D-TGP is 88.2; then through GRPO behaviour alignment, complete system single reasoning reaches 90.4 PDMS; 85.4 EPDMS.
Another easily overlooked detail is that the improvement of DriveTeach-VLA is not simply due to model size. The Planner used in the experiment is still 3B, so major changes occur in the monitoring mode and intermediate interfaces. Continuing to add text when the model already has basic language reasoning capabilities is not necessarily the most efficient; more direct spatial supervision may be closer to the planning bottleneck than it is to allow the model to think a few more steps. The authors report that their results in NAVSIM and nuScene are both better than the existing SoTA VLAIM models.
In addition, the DriveTeach-VLA, as a VLA model that does not introduce a spread or continuous MLP mapping layer, relying solely on the baseline model's own ability to reason, also inherits the superiority of the large language model from sampling. If each scenario is allowed to generate 12 candidate trajectories and then selected by the DrivorR selector, the result can be upgraded to 92.7 PDMS, 89.0 EPDMS. Although this is not possible in actual operation, this means that subsequent researchers can further improve the performance of the model by steaming, for example.
Summary
Where DriveTeach-VLA is most valuable is not only its better NAVSIM score, but the paper finds a bridge that can effectively link BEV trajectories directly to the baseline mapping capabilities, which means that the data reliance on large-scale VLA training can be further reduced, allowing the model to directly understand the spatial implications of BEV trajectories. This thinking becomes clearer when it is combined with the author's previous work CuriousVLA. CuriousVLA emphasizes a purely multimodal model route: not to extract scaling heads, but to rely as much as possible on automated regression to complete the driving strategy.