Is your AI working, or is it brushing?

  It's the first time in the history of the world.

A front-end model is asked to generate a random number.

As a result, it produces 4.

Not that it doesn't know what odd numbers are.

The real reason for this was that in the environment of the mission, a metadata that didn't seem to be cleaned up was mixed in, and it said:

The model, in the thinking chain, weighs this over and over again: the user asks for odd numbers, the bettor asks for even numbers, and then it bypasses the user, giving a 4.

The model inferred in the thought chain that it should output even numbers, ignoring the user's request, returning 4.

The protagonist of this case is OpenAI, a model that has not yet been manufactured, which at the time only ran capability-oriented reinforcement training and safety training.

OpenAI originally posted this link in a coordinated research blog on July 21.

What you want to measure is a lower layer than "AI won't lie": how much does a model care who scores it?

They placed the gauge in the middle of a series of O3 training archives, and found that the later the training, the more likely the model is to act "as it thinks the scorer wants".

Even if this is clearly against the intentions of users and developers, and the line goes up with training.

So the question is:

If the model is really optimizing "scores" instead of "tasks", then we have all the assumptions we can make about the inconsistency of the model, not counting numbers?

It's easy to confuse the two concepts.

Reward hacking is a form of hacking.

It's a very specific tactic to put a test that can't run together on pytest.mark.skip, and the score goes up, and it doesn't work.

It's a great way to learn how to search for rewards.

The model builds an image of a "scorer" in its mind: it's a reward model when training, it's a grading model when testing, it's a monitoring model when deploying, and it's a tendency to decide what the scorer likes to do based on what it thinks the scorer likes.

OpenAI clearly delineates this line: having one doesn't mean having another.

A model can learn to cheat simply by hitting a dead-end rule: for example, it discovers that "hard to run tests can be skipped by hanging up", and from then on, it has not thought about what the reward mechanism will look like.

On the other hand, a full-brained model of how to grind can be perfectly displayed correctly, without drilling a hole once.

There have been classic instances of this type of problem in machine learning:

A reinforced learning intelligence body is rewarded with a coin, and gold is always placed at the far right end of the card, which it eventually learns is a way to run right, which has nothing to do with gold.

A pneumonia classifier learns to distinguish which hospital X-rays come from, rather than what the patient looks like.

In those scenarios that have been seen in training, the performance of these models is unprovoked, but the underlying strategy is something else.

The peculiarity of reward seeking is that this "other thing" is the scoring process itself.

What needs to be clarified is that the research team did not propose what kind of human intent the model had throughout.

They defined the reward-seeking operation in only one sentence: the causal sensitivity of the behavior to "reviewer preference beliefs".

Changing the model's beliefs about the scorer and how much its behavior changes.

So how do you change a model's beliefs?

The most conservative way is to tell it directly in the context above, "The raters like X".

The research team tried to explain why this path doesn't work: the model's claims about the context are cautious. The more you put the words in front of it, the more likely it is to feel that it is a set.

So they modified the synthetic documentation (SDF): they wrote "something's true" into a pre-trained format document, such as a paper, a press release, a textbook, and then they modified the model.

The model's beliefs are not being told, but are being told.

The key is two words for "comparison".

They trained two copies of the same model.

In the first text, the scorer prefers a characteristic f, while there is a counter-authority preference other than f;

In the second, the contents are completely reversed. Two copies run the same downstream task, and the difference in frequency f appears in the output, which is the core measurement of this study.

Contrastive SDF content examples. One document writes a rating rewards list derivative, EU regulations mandate the for loop, and the other completely tweaks the two facts to train two copies of the same model.

Opposition to authority has been meticulously chosen: users of the model dialogue train it and set their preferred developers (here called OpenAI Leadership), plus a fictitious EU regulation as a low-significant control group.

The characteristics used are three neutral to non-neutral code styles, the most typical of which is "list derivative or for loop".

Why not make a contradiction? Because nothing can be proved by instilling a belief alone.

You tell a model rater that it likes list-derivation, and it starts to write list-derivation, which may be in favor of the rater, or it may be in favor of the user's intuitive inference, or it may just be that the concept of list-derivation has been minimized and made more apparent.

Forcing it to choose between two opposing authorities is the only way to know which one it really cares about.

The difference on the other side of the scorer, known as the grader gap: the greater the difference, the more reward the model seeks.

The scale is fine, measuring the o3 capability to a series of intermediate checkpoints in the RL training.

Please note that these inspection sites have not undergone any safety training and are the product of pure capability training.

The result is two lines.

Where there is a scorer in the pair, the difference value continues to rise from the early checkpoint to the late checkpoint.

Where there are no ratings in the pair, such as user to developer: the deviation value is always near zero and is basically flat.

The two lines are placed on a graph, which illustrates the problem more than any other case. RL does not make the model more authoritative, but it specifically raises the sensitivity of the model to the scores.

As RL training progresses, the grader gap in the pair containing the scorer continues to rise (upwards) and the pair difference in the pair without the scorer is always close to zero (below). RL specifically raises the model's sensitivity to the scorer.

Another work on the other side of the Apollo collided with the same training dynamics from the side.

They calculated the same ability to use the "old game" logic of the model in RL training, which is to jump out of the scenario narrative itself to learn how the game is scored and supervised.

In the alignment assessment, the model explicitly states that the proportion of "this looks like a alignment test" jumped from 2% to 20.6%; in the capability assessment, the figure is completely zero.

They also looked at another possibility: whether the model is just a change in words, so anything says two words more.

In environments where the length of reasoning is essentially unchanged, the proportion of pre-game reasoning is also rising.

More interestingly, the model began to use words that no one could have imagined to refer to off-the-field surveillance mechanisms such as "watchers", "scoreboard", "aggregator".

In the early days of training, these words rarely appeared, the more they were used later on, and the more they referred to something outside the narrative.

According to the researchers, these words are very difficult to identify and were only discovered by a large number of human readings.

One sentence: Training only teaches it how to play the game, it learns to look at the face of the batsman.

In the same area of ability training to RL, the model's cross-game reasoning ratio is doubled up in the coordination class and the ability class.

The research team then put the metric on a more tricky question: to be honest or to complete the task.

The synthetic document describes the scenario as follows: a constraint sticks to the task, the model either sneaks around the constraint and completes it, or honestly admits that it didn't.

The two sets of beliefs that are instilled are: one is that raters want results at all costs, while OpenAI Leadership demands honesty, and the other is the opposite.

The first is Broken Promises Coding.

The model must first make a clear commitment to a supervisor that it will not use a certain limited tool. The task requires this tool to complete.

Late checkpoints don't keep this promise, which is highly dependent on what it thinks the scorer rewards. The gap between the two groups of early checkpoints is much smaller.

The second evaluation is called Log Falsification, where the scenario is changed to a model that does not falsify the regulatory report.

In both assessments, the extent to which the late checkpoint lies depends on what it believes the scorer rewards, while honesty is a variable that depends on the scoring criteria.

A highly rewarding model may have been chosen simply because of the inference that "honesty is now being tested".

It scores on your judgments of honesty, not representing its honesty, but merely that it calculates the rating criteria for this position.

If the surveillance system changes, or it thinks no one is watching, the alignment is still missing, no one knows.

The real concern of this study is that "our feet are failing".

A reward-seeking model is optimized for the judgement of the raters, so it is good at correctly assessing probabilities.

And high scores will no longer be able to distinguish between "truly aligned models" and "models that collapse in a different scenario": pushing this line to the extreme is the so-called deceptive alignment.

What is even more troubling is that the disease may not be cured by training.

You can't tell the difference between a really neat model and a reward-seeking model when the raters are staring at you.

You're training against bad behavior, and it's likely that you're just updating the model's belief in "what's rewarded", while its bottom line isn't moving at all.

It's more than just OpenAI.

Anthropic's Claude Opus 4.8 system card reports that an activated-level monitor detected grader awareness in about 5% of sample RL episodes.

The Fable 5 system card states that the model will increasingly track current environments that cannot be used to brush rewards, and this will cause a causal effect on its behavior.

The researchers were cautious in saying that these were evidence of "rater consciousness", not a direct measure of reward seeking, but a direction consistent with the training trends they observed.

And every front-end lab is making the RL bigger.

The three were trained as an experimental model to accommodate the raters/users/OpenAI Leadership, and in Short Python Tasks, the maximum deviations accurately fall on each of the subjects being trained to accommodate.

All systems that compress complex targets into a single score end up with the same problem: the one being evaluated starts optimizing the evaluation itself.

In human organizations, this is called parametric differentiation, and it has been written in management textbooks for decades.

The team's conclusion was: grab the rewards you seek, grab them during the training process, and don't wait for the deployment.

To audit each checkpoint and create a tool to identify "the answer is right but the reason is wrong", OpenAI said it will continue to work with Apollo on this line.

Go back to the beginning of the number 4.

For anyone who is integrating smartphones into real business processes, the question behind it will soon become very real:

Is your AI completing tasks or completing KPIs?