UrbanGround put the big model in a city sandbox built with real three-dimensional data from Hong Kong. The highest success rate for short-range navigation was 75.0%, while the long-range was only 0% to 3.8%. The model was like "take two steps and forget the way": just knowing where to go, turning several routes, the judgments ahead could not be made.
UrbanGround is based on real three-dimensional geographic data from Hong Kong and supports first-person exploration, global maps, day and night weather switching, and pedestrian movement.
Today's large models are no longer difficult to identify landmarks in urban street scenes. In UrbanGround, the visual recognition accuracy of 10 models ranges from 75.0% to 93.8%.
In short-range navigation, the best-performing models can complete 75.0% of the tasks. After the route is lengthened, the success rate for 10 models falls between 0% and 3.8%.
To figure out where this discrepancy comes from, a team of researchers from the Shanghai Transport University, the National University of Singapore, the United Nations, the Chinese University of Hong Kong, the University of Shanghai and the University of Oxford launched UrbanGround. The team turned real three-dimensional geographic data from Hong Kong into a continuously walkable city sandbox.
Address of the project:https://urbanground.github.io
Link to the paper:https://arxiv.org/abs/2608.27456
Huggingface：
Code address:https://github.com/UrbanGround/UrbanGround
Link to the app:
This time, the model not only needs to see the street scene, but also to walk the road.
In past urban spatial intelligence assessments, many stops were on single-street views or aerial photographs. Models only had to answer immediate questions without really taking the next step. Even in navigation, the viewpoints often switched along predefined nodes, and the model did not really find a step-by-step path in a continuous city.
UrbanGround consists of a geographic layer, a simulation layer and a smart body layer.
The 3D Visualization Map and 3D Pedestrian Network, released by the Hong Kong Geospatial Directorate, coordinate geographical coordinates, three-dimensional grids and walking networks into a single set of sandbox coordinates.
The simulation layer transforms the map into a walkable city. Buildings, walls, slopes, and terrain ups and downs affect the operation. The same location can switch day and night, the road can be closed, and pedestrians can move along the walking network. Researchers can run the same route repeatedly, only changing the weather, the closures or the flow of people, to see how the model responds.
The intelligent-body layer specifies what the model can see and do at each step. It can only see the current first-person image and task description, and can open the map when needed, then select to move, turn, jump or operate the map. The map marks the current location and destination, but not its route planning.
Figure 2 The same place can switch day and night and weather, pedestrians move along the walking network and collide with the model.
I'll see what the street looks like, why can't I just walk down the street?
UrbanGround has set up dozens of tasks, but the question is the same: what is the model's judgement on whether to use it after turning, walking away, or encountering a lock?
The paper examines three aspects of spatial positioning, continuous memory and dynamic adjustment. Spatial positioning asks whether the model understands the spatial relationship in front of it. Identifying landmarks is only the first step, it must also know in which direction the landmark is in order to decide where to go; continuous memory extends the path.
The team eventually assembled 810 man-made mission instances covering multiple urban areas of Hong Kong and tested 10 multi-modal models using the same set of action interfaces. The task ranged from short to long: first seeing the eye, then taking judgments along a longer path, and finally dealing with changes along the way.
Figure 3 810 Examples of missions in various urban areas of Hong Kong
Space Positioning: Identifying landmarks but not identifying directions
The first group of experiments looked at only the shortest part of the action. The visual recognition test model can deny the surrounding object, directional understanding requires it to judge the direction of the landmark relative to its own direction, while active exploration allows it to go a few steps further, completing clues that are missing in the initial perspective.
The visual recognition accuracy of 10 models ranges from 75.0% to 93.8%, and the active exploration accuracy from 46.3% to 82.5%. As long as the clue is nearby, the model can often look, walk, and complete the task.
The direction is much more difficult. The directional accuracy of the 10 models ranges from 23.3% to 58.3%. The visual recognition accuracy of the Gemini-3.1-Pro is 82.5%, while the directional accuracy is only 23.3%, even lower than random guesses.
Table 1 Accuracy of visual identification. PNA indicates the ratio of the trajectory to the time of movement on a registered pedestrian network.
The answer is yes, and it's not the same as being right. Some models find the right clue, but the trajectory leaves the pedestrian network. The paper has a very intuitive example: GPT-5.5 needs to answer which bank is next to the Beijing Symphony Hall.
Figure 4 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.5 GPT-5.
Continuing Memory: Two Steps and Forgetting the Path
The second group of experiments extended the route, including language instructions, path limitations, location search and multi-site planning. The form varies, but the difficulty is the same: after the landmark leaves the field of view, can the model still decide where and where to go?
GPT-5.5 and Claude-Opus-4.6 had a success rate of 75.0% in the short run. The success rate of the first to the long run of 10 models was only 0% to 3.8% in the long run. GPT-5.5 fell from 75.0% to 0 and Claude-Opus-4.6 fell from 75.0% to 1.3% in the short run.
The problem is not just that the road is longer, the steps are more. Every time the model turns, crosses the street or switches the map, it has to re-judge where it is, where the target is. It can go right a few steps in a row, or it can still go in the same direction after a turn. Small deviations accumulate and it is difficult to get back on the right path in the end.
Table 2 The highest success rate for short-range navigation is 75.0%, and all models in long-range navigation do not exceed 3.8%. Eight class missions use the same set of city interfaces.
The trend is clear: the longer the distance, the lower the overall success rate of short-range navigation and instructional navigation. The curves of individual models will be upside down, so it cannot be said that performance will decrease with each step.
Figure 5 In the same class of tasks, the overall success rate decreases as the interaction distance increases, but the curves of the models are not entirely consistent.
It is easy to think that the model is wrong at the beginning, if only to see if it reaches the end. The trajectory is not the same. At the end of long-distance tasks, models still have trajectories of 52.5% to 81.3% closer to the target than the starting point, with a complete success rate of only 0% to 3.8%.
In long-range navigation, the end of the trajectory is often closer to the target than the start, and multi-site planning can often complete partial visits, but the overall success rate is still close to zero.
One of the trajectories of GPT-5.5 is representative. Following the green line, the model finds a formal slope, climbs the complex structure of the bridge, and continues towards the target. At least in this section, it understands the road structure and chooses the right path.
In Figure 7, the model finds a formal slope and crosses the bridge.
Another trajectory is problematic at key nodes. The model can identify paths and adjust directions without re-planning after deviation, and errors continue to lead to the end of the mission. Of the long-range GPT-5.5 missions, 57.5% of the trajectory ends up closer to the target than the starting point, and 34.6% of the trajectories shorten the distance by at least 20%. However, when these trajectories are put together, the average remaining distance at the end of the mission is still 98.9% of the initial distance, reaching a rate of almost 0%.
Figure 8 GPT-5.5 Ahead of the target crossing the road, after encountering a central obstacle, it was repeatedly forward until the 100th step was rediscovered.
Dynamic Adjustment: Cities have changed, models are still going the old way
When it comes to dynamic environments, the situation is one more layer: the road conditions have changed, the model will give up the original way of walking? The last set of experiments joined the enclosures and mobile pedestrians during the model operation, to see if it changes path, avoidance.
Researchers switch time and weather at the same location, on the same mission. Local responses are fluctuating, but short-range navigation is not generally different. For example, in GPT-5.5, the short-range success rate for clear days, nights, and dark days is 75.0%, 75.0%, and 76.3% respectively. This indicates that long-range failure cannot simply be attributed to dark or bad weather.
Table 3 Local queries and short-range navigation results from the same mission at different times and in different weather conditions
Then, the researchers went to the model to half-hour closure and put in the mobile pedestrian. In the closure mission, the network of the models followed a rate of 93.0% to 96.4%, the closure compliance rate was only 10.0% to 46.7%, and the final reach rate was only 0% to 3.3%. In the mobile pedestrian mission, the network follow rate was still 90.6% to 98.5%, but the collision rate was as high as 76.3% to 90.0%. These models were still on the network most of the time, but they rarely changed routes due to the front closure and often did not avoid the pedestrian.
Table 4 models are still registered on the pedestrian network for most of the time, but the clock compliance rate is low and the pedestrian collision rate is high.
They'll recognize the landmarks, they'll walk a few steps, why is it all messed up?
Putting these results together, the most remarkable thing is not that the model "didn't do well", but the same judgement about whether it could continue to be used. The short-range navigation success rate was as high as 75.0%, the route was long, but fell from 0% to 3.8%. The problem was even more apparent after encountering roadblocks and pedestrians: the model was still on the road for most of the time, and the roadblock compliance rate was only 10.0% to 46.7%, and the collision rate was 76.3% to 90.0%.
This is also where "to walk two steps and forget the way" really points, and the model is not completely unrecognizable. The judgments made at the first moment often do not depend on the movements at the back. The longer the route, the more changes along the way, the more obvious the situation is that the front and the back cannot be followed.
It's not enough to see where the problem is, based on a single image or a pre-set node. The model must go through a continuous city on its own to see where it starts and why it can't turn around.
Why do you want to do a repeatable experiment in real cities?
The cost of experimenting with robots in real cities is high, and it is difficult for each model to experience exactly the same situation. UrbanGround moves these types of tests into repeatable city sandboxes. The same route can be run over and over, and can only change the weather, traffic or traffic. Before the robot actually enters the city or park, researchers can find out where it is most likely to drift.
The street view in the sandbox, the three-dimensional geometry, and each step's movement match each other. Every step of the model, the next image is updated according to the geometry of the same city.
The paper, the project's homepage, the code, and the App are all open to the public. Readers can walk the streets of Hong Kong as if they were playing an open world game, or access their own Agent to see how it works in a real city structure.