In recent years, as the ability of big language models to reason and use tools has steadily improved, the Deep Search Agent has gained a certain degree of autonomous search and information integration capability. More and more studies have begun to attempt to allow the Agent to autonomously plan search paths to address complex problems that require multiple sources of information.
But when Deep Search actually enters the real world, the research process becomes much more complex.
Information in the real network is scattered across different websites and resources, possibly hidden in a set of videos, a dialog in a podcast, a chart of academic papers, or in a data file that needs to be downloaded to be read. Even after finding the relevant website, Agents may need to continue to operate interactive pages, adjust query parameters, and get the information they really need.
The existing Deep Search research has made considerable progress in many areas such as multiroute retrieval, cross-source reasoning, tool calling, and web interaction, and related Benchmarks have assessed these capabilities from different angles. However, in the face of a real open network environment where multimodal information, asymmetric resource formats and interactive access needs are intertwined, Agent's current end-to-end research capabilities still lack adequate evaluation.
This also raises the question of whether an AI Agent can independently determine what to look for, where to look for, how to access and process the next step when the location of the evidence and the method of obtaining it are not predetermined, and adjust the direction of the search according to existing information, sequencing the evidence from different sources, ultimately identifying a complex problem.
To address this issue, a team of researchers from Hong Kong University of Science and Technology, Guangzhou University, Purdue University, Washington University, Imperial Institute, Stanford University, etc. has proposed a new deep search task paradigm in their paper, OpenDS: Open-Web Deep Search, and constructed an evaluation benchmark for the real open web environment, OpenDS-World.
Deep search in real-world web environments
Link to the paper:https://arxiv.org/pdf/2610.30181v1
It's a bit of a mystery.
Hugging Face links to:
The team built 200 missions, covering 4 modules, 42 resource formats and 7 types of evidence access requirements, giving GPT-6 Astra, Claude Opus 5.5, Gemini 3.8 Flash, Qwen 3.8 Max, GLM-5.3 Flash, DeepSeek V4.1-Flash, Kimi K3 and 12 other front-end models the same set of searches, browsers and video related tools, while limiting 1 hour of search time and 100 tool calls per question, allowing them to decide the search process themselves, sequence evidence, and ultimately provide answers to questions.
The best GPT-6 Astra results were 39%, Claude Opus 5.5 and Gemini 3.8 Flash scored only 38.0% and 36.5%, while other models scored less than 35%. It is worth noting that, although the average source item for each question is only 3.94, a search attempt on the model involves dozens or even close to a hundred tool calls. The failure of the search process is also quite time consuming: in the GPT-6 Astra execution record, the failure mode is only 6.34%, but it consumes 42.57% of the tool execution time.
Figure 1:12 demonstrates the performance of a model on OpenDS-World.
In a deep search, what do we want the model to accomplish?
Researchers are familiar with the process: initially, only one obscure object of interest is searched, and after an introduction, only one report is searched, and a diagram of the report leads the question to another set of data.
Deep Search hopes that Agent will be able to perform such searches autonomously. OpenDS is concerned that when this process occurs in a real open network environment, Agent will be able to autonomously discover and access information scattered across different sources, handle a variety of resource formats and interaction requests, and take advantage of new leads to decide on further search directions.
This autonomous decision-making is reflected in the actual process of accessing the information. Even after finding the relevant website, the agent often needs to take several steps to obtain the target information. The paper, for example, has both text, maps and graphs on the same page, and also provides data resources such as CSV, JSON and ZIP. To query the number of selections in a given region in a given year, the agent may need to adjust the region, time, re-read the graph, or receive messages through data access and file downloads.
Figure 2: The same site can contain both text and graphs, multiple data formats, and operations such as parameter selection, time positioning and data query.
Different sources of information also have their own access and processing requirements. The clues in the video need to be located in specific segments, the evidence in the paper may be hidden in a graph, and the data files need to find the target records and fields. Even when storing text data, HTML, PDF, Parquet are not read the same way.
The paper categorizes these differences into three dimensions: diverse modular information (Diverse Modalities), heterogeneous formats (Heterogeneous Formats), interactive evidence access (Interactive Access Requirements). Modular descriptive evidence is presented in text, images, audio, or video; format describes how it is stored in files and data structures; and access requirements describe what operations it needs to perform.
These three dimensions are combined in different forms in a single search. The objects identified by the model from the video can be the keyword for the retrieval of the paper; new clues in the paper may need to be queried on another website.
OpenDS defines this process as a question-and-answer process with a unique, verifiable answer. After the model gets the question and the initial clue, it needs to find, access, process, and associate every step of the evidence on its own, and finally provide the answer. In this way, the search process preserves the complexity of the true open network, focusing on clear nuclear testing criteria.
How to turn real web searches into an issue?
Two problems must first be solved: each node in the chain of evidence and its associations must have a real, verifiable basis; and the final problem cannot directly expose the intermediate entity or the answer, so that the model can only find the endpoint based on the clues of the problem.
OpenDS-World uses the "build a chain of evidence first, and then reverse generate a problem" approach, dividing the task into three phases:
The research team selected 500 candidate videos from eight fields of culture, everyday life, science, economics, history, nature, society, and technology. The taggers combined continuous video images and audio content to identify the initial entity with a clear orientation and ensure that it was the only one that matched a valid Wikidata entry.
After the confirmation of the starting point, the Open-Web Evidence Expansion is underway. Taggers search for candidate information sources in the real network through Google Search based on current entity-built queries and examine their contents one by one, then select sources that are clearly semantically linked and verifiable, and extract new entities from them as the next search node. The entire expansion process especially emphasizes reliable synchronicity between the evidence. Each new node, although coming from a specific source, needs to be validated through multiple independent source cross-checks.
Finally, the Question Inversion was written. The team set the end of the chain of evidence as a reference answer, and then converted the identified intermediate entities into a set of interdependent indirect descriptions and constraints.
For example, the title does not directly give the name of a museum exhibition, but instead requires that the organism be identified on the basis of the video, then find the corresponding collection, and identify the target fossil from the designated gallery. The first step of the discovery provides the necessary clue to the next step, with the constraints gradually narrowing the scope of the candidate and ultimately identifying the only answer.
Figure 3: An example from OpenDS-World. This problem will link audiovisual identification, virtual tourist navigation, document inspection and structured data retrieval across multiple network sources. From the initial video, through multiple sources such as museum virtual guides, podcasts, academic papers, and structured data sets, the unique number of UNESCO Geological Parks is finally determined.
Such a design makes it impossible for the model to search for the final answer based solely on the name of an entity given directly in the subject, but requires the gradual identification of the correspondence between the intermediate objects to be searched in a real network.
Following the completion of the thesis, the research team organized a total of 10 independent auditors to verify the candidate's tasks in two ways:
Evidence Validity: checks the relationship between the initial entity, the intermediate node, the node, and whether the final answer is true, verifiable, and cross-checked through multiple independent network sources.
Task Validity: Check the clarity of the problem statement, the accuracy and consistency of the constraints, and whether these constraints can together determine the only factual answers.
Candidate missions that contain factual errors, insufficient evidence, constrain conflict or ambiguity will be eliminated, ultimately forming 200 pathways with clear reference answers and verifiable evidence pathways.
From the standpoint of single-track tasks, 82.5% of the topics involve at least two information modules, 96.0% involve at least two resource formats, 95.5% involve at least two types of access operations. Of these, 67.0% involve page or content leaps, 60.0% involve resource transfers, 47.5% involve time positioning, and 42.0% involve parametric status updates. Models in the same research task often require continuous processing of different types of information and access operations.
Figure 4: OpenDS-World's information model, resource format, and access needs distribution. Different categories can overlap in the same task, statistically excluding starting videos provided separately.
How long did it take humans? How much did the model answer?
In order to understand the actual difficulty of finding the answers from scratch, the team organized 50 participants to independently ask questions. Each of them was assigned 20 questions, each of which was attempted by 5 people individually and received a total of 1,000 answers.
Participants can only get original videos and themes, cannot use LLM tools, can only use conventional browsers and searches, and cannot communicate with each other. A single-question time limit for human tests is not set, and participants can end the experiment when they feel they cannot continue to make effective progress.
The 1,000 responses took about 646 hours to complete, and only 211 of them were answered correctly. Of the 200 questions, only 115 were solved by at least one person, and 85 were not solved in five separate attempts; only 29 were answered yes by at least three people.
Figure 5: Successful human independence, time-consuming and subjective difficulty. In the middle, a statistical chart shows the total time each participant spent completing their assigned 20 questions.
The team divided the difficulty of the task accordingly: at least 54 pairs of responses were entitled Low, 61 pairs of responses were entitled Medium, and 85 pairs of responses were entitled High.
For model evaluation, Harness:16 uses a unified tool that provides search, browser interaction, and video and subtitle reading capabilities. Other tools support web page clicking, entering, scrolling, scraping, downloading content, and checking video clips. All 12 models use the same configuration, self-selecting information sources, and operating sequences.
Unlike human responses, which do not have a time limit, models are allowed a maximum of 3,600 seconds of search time, 100 tool calls. These two budgets stop continuing search once they are exhausted. If a complete answer has not been formed, the system enters the final stage of disabling the tool, allowing the model to respond on the basis of evidence. When the response is completed, Qwen 3.8-27B is used to control reference responses, check semantic equivalence and completeness, evaluation frameworks while recording task time and token cancellation.
Figure 6:12 shows the overall performance of the evaluation model and the performance under different difficulty levels. In brackets, T represents the pure-text model.
Overall, the performance of the 12 models on OpenDS-World remains wide. GPT-6 Astra takes first place with an accuracy of 39.0%, followed by Claude Opus 5.5 and Gemini 3.8 Flash at 38.0% and 36.5%, respectively, while the accuracy of the remaining models ranges between 12.0% and 33.5%.
As the difficulty of the task increased, the overall performance of the model decreased significantly. The average accuracy of the 12 models in the three difficulty groups Low, Medium and High was 39.66%, 23.50% and 16.76% respectively. The advantages of the different models also changed: GPT-6 Astra led the low difficulty group, ranked first in the medium difficulty group along with Claude Opus 5.5; and Gemini 3.8 Flash achieved the highest accuracy of 28.24% among the five human participants who failed to solve the difficulty group.
Reference paths are short and actual searches are long
With the results, the next question that needs to be analyzed in depth is: where specifically are these tasks difficult?
As we mentioned at the beginning of this article, the average source entries for each subject that are labelled and audited are only 3.94, but a model's search attempt involves dozens or even close to a hundred tool calls. Even if only three source labels are designed, the median number of tool calls is 74 times.
In retrospect, OpenDS-World's mission presents a clear feature of long-term decision-making (Long-Horizon Decision-Making). The model needs to identify clues in the initial video, by repeatedly searching, cross-page navigation, interactive access, and multi-modal information processing, to gradually find and associate scattered evidence on the real network. The same source may also require multi-step operations: access to the site, finding inputs, setting parameters, and reading results, each of which occupies tool calls. There may also be unrelated candidate sources, unfinished operations, and clues that need to be back-verified.
The team's analysis found that the length of execution did not increase as the size of the evidence chain increased monotonously. Tasks containing 3, 4, 5 and at least 6 labeling sources, the median number of calls to the tool was 74, 56, 72 and 82.5 times, respectively.
Figure 7: The corresponding tool call distribution for different data chain sizes, as well as the average execution length of each model.
The length of the chain of evidence describes which materials support the answers, and the search trajectory records the exploration process that the Agent has undergone to discover, access, and associate this information. OpenDS-World combines the two to reveal the complexity of execution that is difficult to fully measure based solely on the length of the chain of evidence or the number of logical jumps, and highlights the importance of contextual management and adaptation tools for long-term planning in real-world network research.
What is the use of time and tools?
Different models show markedly different search inputs and tool use strategies in the same environment.
GPT-6 Astra averaged 51.05 calls per key to the tool, taking 18.66 minutes, with an accuracy of 39.0%; GPT-5.6 Sol averaged just 8.99 minutes, with an accuracy of 28.0%; Gemini 3.8 Flash's total Token consumption was about 5.22 times that of GPT-6 Astra, with an accuracy of 36.5%. More tool calls, longer execution times and higher Token consumption did not necessarily lead to better search performance.
The differences between the models are equally apparent. GPT-5.6 Sol is more likely to use search tools, accounting for 74.5% of all calls; Claude Opus 5.5 and Qwen 3.8-Flash are more dependent on browser interaction, with GUI calls accounting for 51.5% and 49.4% respectively, while the accuracy of the two shows a gap of 26 percentage points.
Figure 8. Distribution using different modeling tools
The study also found that the model's tooling calls also showed a clear pattern of continuous execution of similar operations. For example, the Search → Search transfer ratio for Gemini 3.8 Flash was 84.6%, and the GUI → GUI transfer ratio for Qwen 3.8-Flash was 85%.
Figure 9. Category transfers between adjacent tool calls of different models
According to the results of the paper,The frequency and category distribution used by the tool does not fully explain the differences in performance of the model.In real Deep Search, effective coordination of searches, web interaction and information processing, and the full use of the evidence obtained, are important factors in affecting mission performance.
Effects of Failure to Call
Deep in the execution records, the team discovered an expense that could easily be overshadowed by the number of times: tool failure.
The number of failures in each model is equal to the number of failures in each model.2.83% ~ 9.42%The corresponding time consumes the cumulative time of the tool's execution.19.27% ~ 61.90%In the example of GPT-6 Astra, 6.34% of calls failed, accounting for 42.57% of instrument execution time.
In such a task that not only limits the number of calls the tool makes but also the time it takes to complete, even fewer calls fail, the costly waste of time directly occupies the space for further exploration.How to reduce inefficiencies and improve the efficiency of failure recovery has also become a focus of concrete optimization.。
Figure 10. Frequency and time of failure calls in the model, and choice of next steps after failure.
The paper also analyzes the decisions made by models after the tool fails.Direct switching tools are the most common choice for most models.Claude Opus 5.5, Gemini 3.8 Flash and GPT-5.6 Sol had a failure rate of 83%, 81% and 79%, respectively; Kimi K3 and GLM 5.3 had a higher percentage of parameters re-tested on the same tool at about 38% and 37%, respectively.
Did you really find out at the end of the search?
For the Deep Search Agent, when to end the search is also an important decision. Both the model needs to complete the research within a limited budget and determine whether the information currently obtained is sufficient to answer the question.
OpenDS-World submits a complete, non-empty answer to the model, and no longer requests the tool as a natural ending.69.0%While the mission can naturally end the search phase, its ultimate accuracy is only limited.39.0%。The ability of a model to actively complete a search does not mean that it has received sufficient and correct evidence.
Another scenario is that the model continues to operate until the budget is used. Qwen 3.8-Flash has 80.0% of the tasks terminated by reaching the call limit, while DeepSeek-V4.1-Flash has 60.5%; GLM-5.3 has 41.0% of the tasks touched only the time limit and did not reach the tool call limit. There are also significant differences in the main budget constraints faced by different models.
Even with the relatively low-budget GPT-6 Astra and Claude Opus 5.5, the overall accuracy remains below 40%.
Figure 11. Reasons why the model stopped searching
OpenDS-World thus reveals two core challenges in real deep web search:How to complete a search on a limited budget and how to give the correct answer at the end of the search.In addition to improving execution efficiency, effective information source selection, evidence integration and search stop decisions, they are also important directions for enhancing Agent research capabilities.
The next step is to keep improving from the search experience.
At the end of an in-depth search, the Agent will leave a complete execution trajectory, including tool calls, interactive feedback, failed operations, and resource consumption. These feedback from real-world network environments provide a specific research cut-off point for improving the Agent's search strategy.
As the paper put it in the discussion section,OpenDS-World can be a natural experimental environment for the study of recursive self-improvement (RSI).
For example, in the face of repeatedly failing web operations, can the Agent identify problems from historical trajectories, adjust tool selection and recovery strategies? In the face of complex long-range search tasks, can it summarize more effective sources of information discovery and resource allocation methods to achieve better results on a limited budget?
Figure 12. The tools of the different models, which call for budget depletion, tool failure rates and failure time ratios, present potential improvements in Agent execution strategies.
The search results on the real Internet, the structure of web pages and the state of resources are constantly changing.Agents need to gradually develop research strategies that are able to adapt to different sources of information, access mechanisms and task needs, and test their effectiveness through actual feedback in new tasks.
OpenDS-World combines complex search tasks, verifiable result feedback, and complete execution trajectories across real open networks to provide a foundation for the study of Agent search, interaction, and long-term decision optimization.
When the Agent is able to independently conduct complex network searches, accumulate them in real-world interactions, and translate these experiences into more reliable responses to new tasks, perhaps it is the next important threshold that the AI Agent needs to cross.
© THE