Is the driverless world so crazy?
In addition to being tested on real roads, unmanned vehicles can now enter AI-built virtual cities, completing 100,000 kilometers of virtual roads per day.
Here, the front vehicle may suddenly brake, the next vehicle may always be ready to align, and pedestrians may appear from the shadows. Unmanned vehicles observe the surroundings to determine what to do next. Once it slows down, accelerates or changes course, the surrounding vehicle and pedestrians will change their behavior as well.
On 3 November, Baidu Apollo unveiled its Apollo World model to the world for the first time, allowing unmanned vehicles to continuously interact with AI-powered traffic participants in a virtual environment, repeatedly experiencing different driving scenarios.
According to industry analysts, Apollo World can measure up to 100,000 kilometres of virtual roads per day. In turn, one day of virtual testing is equivalent to three years of continuous testing of a real roadside vehicle equipped with security personnel.
This is the equivalent of having an AI trainer and an examiner who can change their subjects at any time.
Apollo World is responsible for building virtual cities, constantly changing the road environment, adjusting traffic flow and traffic participants' behaviour, designing different driving tasks for a fast-paced end-to-end driving system. Unmanned aerial vehicles can be repeatedly practiced and tested in the face of changing conditions, judging, taking action, and adjusting the next steps according to environmental feedback.
As AI begins to teach AI to drive in the world it creates, there are new possibilities for unmanned driving experiences.
The "impossible triangle" of driverless technology
AI needs to break down.
Driving this thing seems to have clear roads, lanes and traffic rules. But for an unmanned vehicle without security personnel to be able to operate on a large scale, the first step must be safety. In addition, the unmanned vehicle must be smart enough to understand the intentions of the traffic participants around it and be able to make reasonable judgments based on the ever-changing road conditions.
In the face of "impossible triangles" such as security, intelligence and scale, real road measurement inputs and speeds are difficult to satisfy. In addition, the long-tail scenario also has a natural difficulty: low frequency occurs. A car runs hundreds of thousands of kilometers and may encounter a large number of repetitive normal road conditions, but does not have to encounter the special interaction that researchers want to measure.
Chief architect of Baidu Intelligent Driving Group (IDG), Chairman of the IDG Technical Committee Dr Wang Liang
Traditional simulations can help unmanned enterprises solve some of the problems. For example, recycled simulations record real road scenes and repeatedly test them for different versions of driving systems. However, the scenes always run on historical tracks, with unmanned vehicles not responding to changes in movement and surrounding vehicles.
This is the place where AI creates driving experiences.
The world model allows AI intelligent agents to participate in it, adjusting the traffic environment in real time based on each movement of the unmanned vehicle. A real route can be depicted in many different traffic situations, giving unmanned vehicles the opportunity to repeatedly try different driving strategies.
In order to create such virtual driving, it is necessary to first understand what kind of model the world needs to be without a driver. In recent years, technology from video generation, 3D scenery construction, to environmental simulation and intelligent body interaction has been dubbed the model of the world. But image realism does not represent a real understanding of the world, or the ability to reproduce scenes, nor does it represent the ability to drive post-interaction changes.
Faced with an increasingly broad concept, Baidu chose to start from the actual need for unmanned aerial vehicles (UAVs) to clarify what requirements the world model needs to meet, and then to guide system design and technical verification. It was also the starting point for Baidu's "2+3" judgement criteria: causality, interaction as two cornerstones, physical accuracy, durability, and real-time as three enhanced attributes, and the ability to define a more precise set of criteria for the UAVs.
Brakes can drive consequences, accelerate changes in the behavior of the surrounding vehicle, test causality and interaction; brake longer distances in rainy weather; maintain consistency in continuous lane position; test physical accuracy and durability; all changes must also keep pace with driving and meet real-time.
The five criteria ultimately point to a challenge: Will the entire world run like a real road when unmanned vehicles enter the virtual city?
AI builds a city
No-man's-car to train
When an unmanned vehicle enters the world of Apollo World, a special "instruction" begins:
Building a self-sustaining virtual city
The first thing that Apollo World needs to do is not only "draw out" the city, but to make it work on its own.
It models vehicles, pedestrians, and others as AI intelligent agents with perceptive and decision-making capabilities, with their own behavioral logics and goals, rather than background roles based on predicted script actions.
The effect is intuitive, and the entire traffic situation may change as a result of a single driving move. The driverless vehicle chooses to go to the railing, and the next vehicle can enter; and if the driverless vehicle chooses to go to the ground road, the next vehicle may stop and wait.
On the same route, the following traffic situations may lead to different outcomes just because the driverless cars are chosen differently. This is behind the Apollo World joint modeling of participants in self-driving and surrounding traffic:
The model receives control signals such as steering wheels, fuel gates, braking, and current scene information to execute the next action and thus generate new traffic conditions as the starting point for the next round of interaction.
More importantly, the environment is not fixed. Engineers can continue to increase the difficulty: adjust the traffic flow a little more tightly, set the surrounding vehicles a little more aggressively, or, in turn, make them more conservative, more compliant with the rules. Thus, the same route, the same starting point, can also evolve completely different traffic pressures.
It can be said that in this virtual city, the face of the unmanned vehicle is a continuous gamble that is closer to the real road.
Let the carrot run into the virtual world
If Apollo World were to start by "running" a virtual city, then let the fast-running carpet "run in" the end-to-end driving system.
Real unmanned systems rely on camera and laser radar sensing. Apollo World must also provide it with a set of "sensory systems" that can be used directly: camera images and laser radar point clouds.
With this set of observations, the driving system can start working as if it were on a real road. It reads camera images and laser radar data, judges the surrounding road conditions, and then produces acceleration, braking, turning and other driving movements. These movements are then inverted into Apollo World, driving the virtual environment to continue to evolve: road status updates, the behavior of surrounding vehicles and pedestrians also changes, and new camera images and dot clouds are generated and returned to the driving system.
The world can even "change its skin" at any time.
The same road can be switched to completely different external environments: tropical rainforests, deserts, polar roads, coastlines, tunnels, and even snow slopes. The road itself remains relatively stable, but the surrounding environment, lighting conditions, atmosphere, and even sensory signals change.
Thus, for an unmanned vehicle, it can repeatedly experience very different driving conditions in similar road structures. The route, which was originally only tested in one environment, turns into a lot of different difficulty terrain. The value is obvious: it gives the driving system more driving experience under different conditions.
In order to produce this world in a stable way, Apollo World uses a multi-modal generation model at the bottom that causes self-return. Simply put, it will push the future forward in chronological order. The state of the world of the previous moment, coupled with the current input driving movements, determines what will happen next. Rather than directly "spelling out" the future, it is continuously evolving a process of causation.
In a more straightforward way, Apollo World allows different sensors to "see" the same future together.
From a road junction, the entire Robotaxi route starts.
It's hard to know how to deal with emergencies at a single intersection without actually learning to drive.
For a Robotaxi, the real task is to get passengers safely from point A to point B. In the middle there may be dozens of intersections, and different vehicles and pedestrians, and it must continue to cope with various changes in traffic.
The world model must remember what happened in the past in order to keep traffic participants and road conditions consistent over a long period of time. This is the important meaning of the world model "sustainability", which needs to generate a future state that is consistent in time and space, so that the state of the surrounding vehicles can take on previous movements and continue to influence subsequent scenarios.
Now, engineers can arrange a whole bunch of Robotaxi tasks for Carpenter: starting from any starting point in a virtual city, traversing different roads and traffic environments, completing the entire process of ordering, picking up passengers, delivering to a destination.
More interestingly, multiple virtual vehicles can also perform different travel missions at the same time, form a virtual fleet, and simultaneously perform road surveys.
This capability is not limited to a specific city or type of road. Test routes also extend to different cities and regions, including beach roads, near airports, Linden Avenue, and around schools, allowing drones to be tested on a large scale in different areas in advance.
Waymo and Baidu
Two ways to create a virtual world
I'm sure everyone has an intuitive sense of what the virtual world is like when Apollo World is built. But when you look at the entire world's model industry, there are different ways and directions for teams to build the virtual world.
Some are good at generating images, representing Sora, which is shut down by OpenAI; others have interactive simulation capabilities, representing Genie 3 from Google DeepMind.
Others focused on building three-dimensional space, with Marble representing Lee's first product, World Labs, which is capable of generating high-fidelity, cool three-dimensional space, and can be freely explored in it. Apollo World went even further in its capability to enable vehicles and pedestrians in the virtual world to be autonomous and interact and respond in real time.
Li Fei Fei Marble exhibition
Similarly focused on the vertical model of the unmanned world, the Waymo World Model is also an important reference in this Apollo World.
However, the two companies have chosen different technology intersections.
The Waymo World Model is based on Google DeepMind's Genie 3 for driverless field adaptation, optimising the Waymo Driver's sensors and driving tasks in addition to the generating capabilities of the general world model.
It is most eye-catching to move scenes that are difficult to encounter on real roads into the virtual world. For example, vehicles encounter tornadoes, roads are flooded, and even animals such as elephants and lions suddenly appear on the roads.
Waymo also provides multiple scenic controls. Engineers can change the weather, time, and surroundings through natural language Prompt, as well as adjusting road layout, traffic signals, and the location and behavior of other road participants.
However, from the publicly displayed material, Waymo's scene interaction still relies heavily on engineer setup and has not yet demonstrated the full order process.
Waymo World Model on the left and Apollo World on the right
While Waymo is good at creating new puzzles for unmanned vehicles, Baidu Apollo World is focusing on something else: letting participants in the virtual world make their own judgments, rather than simply moving along engineers' trajectories.
Another advantage of Apollo World over Waymo is that it allows unmanned aerial vehicles to take "addiction tests" covering the entire mission process.
At the same time, virtual cities need to be real enough that they can not only be generated by models in space, but also have real roads to provide a reference. Carnival has landed in 28 cities around the world, completing more than 23 million journeys, with total unmanned miles exceeding 350 million kilometers, of which the total unmanned miles exceeds 240 million kilometers. These real road driving experiences have accumulated rich scenic material for Apollo World.
Thus, the experience of driving on the real road continues to enter the virtual world, becoming a recurring theme; the problems of virtual test detection, which will be returned to the end of the vehicle, driving the evolution of the driving system. Improved systems continue to put into real operation and bring back new data. The real road and the virtual world are thus constantly interchanging "feeding" to form a continuously running data flyer.
For Robotaxi, which is moving to scale, Apollo World's exploration is more realistic: it takes the world model further into the real R&D chain of unmanned aerial vehicles and transforms the value of virtual testing into continuous advances in the driving system.
The next battle for driverless driving
In the virtual world
If we look back at the entire unmanned industry from the perspective of Apollo World, the changes to the world model may first be reflected in the research and development process.
In the past, when driving systems encountered problems, researchers and developers often had to look for clues in real-life road records and try to reproduce problems, design experiments, verify improvements. For some low-frequency, high-risk scenarios, it may take a long time for problems to reappear.
The world model provides a new solution to this: researchers can actively create test conditions and conduct more control experiments around the same problem. Driving experience that would have to wait for real roads has now been created and accumulated in the virtual world.
The way in which driverless safety verification is done will also change. Numerous repeatable tests and high-risk scenario verification can be completed in advance in the virtual world, with the real road continuing to undertake the final inspection task.
These changes may also affect the pace of commercialisation of unmanned vehicles. Unmanned vehicles entering new cities can allow driving systems to become familiar with the roads, experience complex interactions, and adapt to the exposed problems if they can build a local virtual environment beforehand. Until they actually enter the operational phase, businesses are no longer faced with a completely unfamiliar curriculum and are expected to reduce the costs of early testing and deployment risks.
Putting the vision further, this idea could extend to a broader area of physical AI beyond unmanned aerial vehicles, such as robots. They face similar challenges: how to get more controllable, repeatable training and verification opportunities for the intelligent body before it actually undertakes complex tasks, and how to bring the accumulated experience back to the real world.
This also means that the industrial value of a world model can go far beyond scenario generation. As the model gradually becomes more interactive, predictive, and physically simulated, it has the opportunity to become a layer of infrastructure for physical AI research and development that connects real-world data, training and testing in virtual environments, and ultimate reality testing.
© THE