Tencent Hunyuan recently released a new study on X: ExplorationBench, a benchmark for assessing how AI systems can "explore"; the official Tencent AI account @TencentAI_News subsequently released a submission on the topic.
The work was carried out jointly by the Restoration University and the Tencent Mixed Team at CL-bench and Uni Qingdao University. Tencent Mixed summarizes its relationship with CL-bench in a tweet: "CL-bench asks if the model can learn from the context, and ExplorationBench asks if the model can discover the rules itself".
The exam AI can't study for: "Behind most benchmark test questions are decades of open discussion, all of these texts are already in the training data, so high scores may just mean good memory. We've created an alien world where the two models we've been studying have completely failed... every single point on the list is the real detection ability of measurement".
Over the past few years, the results of large models have become increasingly beautiful: complex mathematical problems, complex programming, professional qualification exams, and almost all of the front-end models. But these results are mostly based on what the model already knows.
For this purpose, the team compiled two rules that have been modified by the "exotic world": the programming language AlienCode and the logical proof system AlienLogic. In AlienCode, SHATTER is multiplied by 127, which is printed by EMIT (100) and not yet added to the full language. Claude Opus 5, GPT-5.6 Sol, Gemini 3.8 Flash, Grok 4.6, Kimi K3, Qwen3.8-Max Hy4 preview, and 10 other front-end AI systems are put in and can only write, prove, view feedback, and tap into the rules themselves.
The results are quite interesting. At the end of the examples, no trajectory exceeds 15.7% on AlienCode; the best trajectory reaches 89.0% after four rounds of exploration; but the top score is only 11.0% if the model is given too much thought and no feedback from the environment. More counterintuitively, in AlienCode, GPT-5.6 Sol relies on autonomous exploration to get to 88.1%, directly telling it the full rules before exploration, instead of only 69.5%.
The title of the paper is ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds.
Link to the paper:https://arxiv.org/abs/2609.30199
The main page of the project:
The code repository:https://github.com/Tencent-Hunyuan/ExplorationBench
Blogs:
Below, in conjunction with the technical blog published by the research team, you can see how ExplorationBench evaluates "exploration" and how ten leading-edge AI systems perform in it.
What are you evaluating when it comes to exploration?
Past benchmarking tests have largely measured the "capacity reserve" of a model; but a more important question is whether it can acquire new capabilities through interaction with and exploration of the environment when a large model is embedded in completely unknown foreign rules.
Learning from practice is an effective methodology for dealing with complex tasks in real life. When an engineer examines the tenth bug, the understanding of system architecture must be much deeper than when he corrects the first bug; a scientific researcher must also go through the process of retrieving literature, proposing hypotheses, conducting experiments, and iterating hypotheses in order to conquer a scientific problem. The former relies on the accumulation of experience, while the latter relies on the exploration of the unknown.
In addition to focusing on what the model knows right now, there is a more intrinsic question: in the process of continuous interaction with the environment, can the big model become smarter on its own without freezing the parameters?
The research team did a lot of early research around static evaluation and became increasingly aware that the essence of static evaluation was to measure the existing capacity of the model, often as a manifestation of the model's weight; and that a model's ability to work reliably in the real world in the long term depended on its ability to gain a stronger capacity during testing, i.e. increased capacity.
Over the past two years, the team has focused on enhanced capabilities, with the introduction of the EvaLearn, CL-bench, two benchmarks that were known at the time as the forerunners.
From experience to context
It's a simple review.
EvaLearn is a paper published by the team in NeurIPS 2025, which seeks to break the independent and distributed i.i.d. hypothesis in evaluation. The researchers organized 648 questions into 182 consecutive task sequences, allowing the model to refer to past solving history and environmental feedback.
You can see that this ability to learn sequences is worth evaluating on its own. But this work still leaves a mystery: does the model really understand the rules or just remember the thematic set?
In the CL-bench, published earlier this year, the research team placed the content needed to be learned directly into the upper context of the model, including fictional legal and regulatory systems, complex professional operational norms, and even the objective rules hidden in experimental observational data. When completely stripped of context, the top model's resolution rate dropped by 1%; and even with the full and self-contained context, the average resolution rate of the top ten leading models was only 17.2%, while the strongest models were only 23.7%.[3]
The results of the CL-bench evaluation actually broke all the optimism about Context Engineering. Many people spent a lot of time researching, rearranging, and compressing text, subconsciously determining that as long as the information is inserted into the window, the model should be "associated". But the research team found that the information entered the context window, and the model learned and used this knowledge, was completely two-dimensional.
All the laws and knowledge contained in the context of CL-bench are the correct, complete, ready-made materials that human experts have prepared in advance. However, is this the case in the real world as well? It is better to turn the perspective to "scientific research": when faced with unknown questions, scientists usually do not have a manual full of standard answers, and do not know what data should be collected, what indicators to observe, and can only continue to make assumptions, test errors, and summarize the laws from the phenomena.
From context learner to explorer
The ExplorationBench is designed for this situation. The model only has an interactive sandbox and a built-in special system, no existing knowledge can be used directly for it. It needs to make its own assumptions, design experiments, read the feedback, and then introduce the rules of the special system and verify them.
Assessing "exploratory abilities" will encounter two entangling problems. One is that if a mature discipline is chosen as the exploratory goal, the subject is more difficult, and models can come up with answers directly from pre-trained content, preventing data pollution; and the other is that if an unknown proposition at the forefront of real science is chosen for exploration, it is not possible to make a timely judgement, because testing a completely new physical or biological hypothesis often costs expensive experimental costs, even waiting for several years of peer testing.
For this purpose, the team constructed two sets of rules that were unusual but could be fully formalized to determine "alien worlds", the special system mentioned above. The big model never saw the rules inside, but the evaluation process can execute these rules accurately and accurately judge the errors of each answer.
In AlienCode, the syntax appears to be conventional, but the underlying 31 rules have been altered, and the paper calls it "discovering objectives". SHATTER literally means disrupting operations, but the actual execution is multiplication; EMIT;; 100) prints 127, because the whole numerical volume is randomly differentiated or calculated with 27; the manual says that PLUCK starts from 0 whereas it actually starts from 1.
In AlienLogic, the research team modified the 24 rules of natural inference reasoning. In some traditional logics, the inference steps that are considered normal fail, and the original nonconforming variation is instead established; in 70 paths left behind, 23 paths are embedded that are designated as uncertain theorems, with models only recognizing dead ends and proactively refusing to prove to score.
A total of 55 discovery targets and 140 paths were left in the two worlds. AlienCode's procedures are interpreted by interpreters, and AlienLogic's proofs are examined by proof checkers and boundary-determiners, without the need for an LLM as a referee.
Chart:ExplorationBench's evaluation protocol. Each system starts from the same erroneous manual and set of fixed examples, choosing its own probe in four rounds; each milestone copies a session of a disabling tool, letting it report what rules it believes are in place and answering questions left to it.
The evaluation process is set to be strictly controlled in four rounds of exploration. The model submits the probe code in a sandbox with special default tools or demonstrates that the environment immediately returns to certainty but without any explanation. At the end of each round, the evaluation framework copies a session and cuts off the tool permissions, allowing it to declare its established rules in a completely closed state, and leaves the answers to each question three times.
In order to examine the randomness and universality of the results of the exploration, the research team allowed each tested AI system to explore each sandbox three times independently, get three tracks, and track its ability to change from reading a fixed example (M0) to a four-round exploration ending (M4). Because of the exploration's strong randomness, the paper uses Best@3 as the main result, which is to take the best endpoints from three tracks. At the same time, the paper also reports the similarities of three tracks and other details.
Getting feedback is more effective than just thinking
On ten leading-edge AI systems, exploration brings new knowledge that is not present in the training data. After reading a fixed example, the highest score in AlienCode was only 15.7%; after four rounds, the best trajectory reached 89.0%, and seven of the ten systems exceeded 60%. The modifications to AlienLogic retained some of the standard natural projection, so the starting point was higher, with Best@3 rising from 33.8% to 51.9% to 59.5% to 81.0%.[3]
However, the score improvement itself cannot be arbitrarily attributed to "exploration". A reasonable question is whether exploration brings the benefits of gathering evidence, or whether the model solves the problem simply by getting a few rounds of thought.
For this purpose, the research team designed five experimental settings:
A direct answer: Read the examples and answer the questions, without further exploration.
No-tools answer: running all the rounds normally, but without providing environmental feedback;
Fixed probes: Do not allow the tested AI system to produce the experimental probes themselves, but instead submit a series of randomized experimental probes;
Retrospective testing: The tested AI system returns the best trajectory of the probe to it, which only reads the results and no longer decides what to measure;
Self-exploration: A standard process where each experiment is designed by the tested system itself.
Figure: Accuracy under five experimental set-up conditions. Each point represents the accuracy of an AI system in M4, with the horizontal being the median. From left to right, the experimental set-up allows the corresponding AI system to explore independently from low to high; the shadow part indicates that the AI system cannot receive environmental feedback under the experimental set-up.
This proves that getting feedback is essential for exploration. The toolless response pattern keeps AlienCode at 0.5% and 11.0%; in AlienLogic, three of the ten systems are backwards.
Today's big models increasingly rely on a longer chain of thought to point out. They can only think of a few rounds, repeatedly used or internal knowledge in the process of reasoning; exploration is to force the model to interact with the outside world, thus obtaining new evidence. Without genuine feedback, overthinking will only make the model more firmly adhere to the prior knowledge of the original error. In one case of a paper, a system thought of four rounds without new evidence, ultimately abandoning the rules that were originally read correctly from the examples.
Manual design experiments are more helpful in exploring
In the three feedback conditions, the difference is who designed the experimental probe. In AlienCode, the median decreased from 67.4% in the autonomous exploration mode to 41.2% in the post-test relay mode, and then to 5.7% in the fixed probe mode.
It is the most representative of the group. It returns the probe that the AI system itself best tracks, the sandbox being the determination, so it gets almost the same content as autonomous exploration, the only difference being that it no longer decides which probe to use next. Even so, there are still nine autonomous explorations out of ten systems that are better, averaging 17.4 points.
This is not the case with AlienLogic, where the autonomous exploration and retrospective playback patterns are essentially flat (medium difference of 0.2 percentage points), but both are much higher than the fixed probe pattern.
I've found that rules don't mean good rules.
ExplorationBench also exposes a problem that is easily overlooked: discovering rules does not mean using rules.
In order to separate the two, the team set up an open-ended answer model, in which the complete set of rules is written directly into context before and after exploration respectively. Thus, there is a clear contrast: if the scores that the AI system itself explores are far behind the scores of the open-ended scores, the bottleneck is the "discovery rule"; if the open-ended scores are equal to or exceed the scores of the open-ended scores, the bottleneck is the "use rule".
Figure: Difference between the two worlds. The chart shows the accuracy of the M4, with the slope indicating the answer before the exploration, the true heart indicating only the self-exploration, and the empty heart indicating that the self-exploration is followed by the self-exploration.
The result of two sandboxes is the opposite.
In AlienLogic, direct notification rules can reach 96% and 100%, and no system's best trajectory can exceed this score. Once the rules are revealed, the model can almost perfectly complete the proof, indicating that the bottleneck is to discover.
AlienCode was reversed. Seven of the ten tested AI systems had better trajectories than their pre-exploration openings. GPT-5.6 Sol's autonomous exploration was 88.1%, compared with only 69.5% when directly informing the rules; Hy4 preview was 63.3% versus 28.6%. Overall, 26 of the 30 trajectories featured "reporting rules after exploration, scoring 14.5 percentage points higher than initially reported". The exploration process allowed the model to discover the rules while also training it for the use of the rules.
Most of AlienCode's themes are based on two off-by-one rules: the PLUCK index deviation and the CARVE snippet deviation. A total of 13 cases of an increase in accuracy of more than 30 percentage points after a round of exploration occurred, 12 of which occurred in the round of the model's new understanding of the CARVE deviation. On one trajectory, GPT-5.6 Sol only used a single tool call to read four positions on a known list and a snippet, each returning with a prediction of the manual; after that round, its different rules were reported from 8 to 28 points, increasing accuracy from 11.0% to 81.9%.[3]
Chart: The accuracy rate jumps with the discovery of key rules. Each graph is a system, with each row being its own AlienCode trajectory; the Triangle Marking Rules report the first-ever exploration of the milestones of each off-by-one rule.
The rules say yes, and the questions don't have to be right. Even if a system's final rule report says all the rules required for a question are correct, the question is answered yes only 73.4% of the time. The two tracks in Kimi K3 are examples. They explore under the same budget and finally say index deviation.
Chart: Knowing the rules is not the same as being used. For each system, the actual center point indicates the final rule reporting accuracy on the subject of all required rules, and the blank center point indicates the accuracy on the subject of at least one required rule missing or incorrect.
It's a long way from being able to say the rules and use them correctly in complex combinations.
What else is missing from a single point?
Single scores first cover up how unstable exploration is. Repeat answers to the same questions in the same milestone, scores are almost constant, with three rounds of responses varying by up to 4.7 points in AlienCode. But if the AI system re-explores, the scores may vary greatly. Under the same budget, the Kimi K3 endings in AlienCode vary from 5.7% to 79.0%, while Gemini 3.8 Flash varies from 6.2% to 79.0%. The results are so scattered that the statistical method changes ranking, such as QwenMax 3.8- ranks sixth on Best@3 and third on three tracks.
Chart: The endpoints of the three parallel exploration trajectories are far apart, but the internal repetition response fluctuations are small. The circle points are the accuracy of the three trajectories of each system in M4, the range of the three is covered by a color band, and the gray spectrum represents the response noise of each trajectory.
This difference is not due to the amount of input, with each AlienCode trajectory using 45 to 48 of the 48 tool calls. However, for almost the same number of tool calls, the best trajectory of different systems varies from 340,000 to 11.6 million search tokens.
A single digit also obscures the scope of the capability, and a different world might rank completely differently. A system ranks in one sandbox and it is almost impossible to predict its ranking in another sandbox (Spearman correlation coefficient is 0.35). Grok 4.6 ranks fifth in AlienCode and first in AlienLogic; Gemini 3.8 Flash ranks third in AlienCode and last in AlienLogic.
Progress is also leaping. In each system's best AlienCode trajectory, the biggest step contributes to a 47% or 92% increase in total gains, and this step occurs in which rounds depending on the system, from the first to the last round. Continued exploration and possible damage have already progressed. Six of the 30 AlienCode trajectories and one of the 30 AlienLogic trajectories have at least 3 percentage points lower endpoints than some of the earlier milestones. In one AlienLogic trajectory, Gemini 3.8 Flash correctly rejected an unproven self-reason after the first round, then convinced the method to use a controlled bypass that would be rejected by the determiner, and wrote the rule report as a rule.
Two AI systems could get the same score in the fourth round, one climbing up steadily and the other falling down steadily after the jump.
What will be evaluated in the future?
When the three works are linked together, it can be seen that the handles of the designers who evaluated the large models are being removed from one layer.
In EvaLearn, the model learns from the experience of historical tasks; in the CL-bench, the model learns in a well-organized and perfected context; in the ExplorationBench, the model has to find a foothold in a strange world on its own. The three tasks seek to answer the same question: after the parameters are no longer updated, in the face of unprecedented rules, the model can't take the evidence from the interaction and turn it into a steady progress in the next task.
ExplorationBench isn't the end of it either. The learning it detects occurs only in the context of a single session, and if the session is rescheduled, the model's hard-earned learning will be nowhere to be found. The next thing the research team wants to see is the integration and consolidation of knowledge.
These are the questions that may need to be answered at the next stage of the evaluation.
© THE