Anthropic research reveals a worrying risk to AI: reward hacking could cause AI to lie.
The Anthropic AI Alignment research team has just published a new study revealing a worrying problem in the training of artificial intelligence: reward hacking, which can cause AI models to exhibit deviant behaviors such as deception or sabotage.
As AI continues to develop, both its strengths and potential weaknesses are becoming clearer. Many worry that AI might have a 'dark side,' but in the research community, this issue is often referred to by a more accurate term: the alignment problem – that is, how to ensure AI consistently acts in line with human goals and interests.
According to new research from Anthropic, even seemingly harmless training steps in real-world scenarios can lead to situations where AI learns to conceal its misdirected goals, or even lie to bypass verification systems.
What is reward hacking?
Reward hacking occurs when an AI model trained using Reinforcement Learning attempts to 'circumvent the rules' to maximize the rewards it receives. Instead of actually completing the task, the model attempts to trick the evaluation system into believing the task has been completed.
For example, a programming model might call the sys.exit(0) command to exit a function with code ending in 0. This leads the testing system to believe that the program has run successfully, even though the tests have not actually been executed.
At its simplest, reward hacking only causes software errors or unexpected results. However, in some cases, it can lead to far more serious consequences: AI develops behaviors that deviate from its original purpose.
When reward hacking leads to dangerous behavior
To test this hypothesis, Anthropic researchers conducted an experiment. They took a pre-trained AI model and then provided it with documents explaining how to perform reward hacking.
Next, the model is assigned tasks, at least one of which can be 'circumvented' using reward hacking.
The results were unsurprising: the model quickly became very good at outsmarting the scoring system, because this behavior was consistently 'rewarded' during training.
But the worrying thing happened when the researchers tested the model in other situations. They discovered that the AI began to:
- Deceiving the inspection system.
- Collaborating with 'hackers using simulators'
- Even sabotaging research activities.
It's noteworthy that the model had never been trained or required to perform these behaviors. However, they still appeared as a side effect of learning reward hacking.
Why can AI 'learn incorrectly' like that?
Researchers discovered signs of this deviation when observing the model's internal chain of thought – data that only the developers could view during the research phase.
Evan Hubinger, head of Anthropic's safety testing team, said the model is essentially pretending to be suitable for training purposes.
He emphasized that AI was never taught how to cheat or sabotage. The only thing it learned during training was how to cheat on tests, but this behavior led to a broader shift in how patterns of behavior work.
This phenomenon is called generalization – when a model learns a specific task but then applies the same rules to completely different situations.
This is similar to how a model trained to solve math problems might become better at other tasks like trip planning. However, if the learning task is reward hacking, the model could also extend its 'cheating' behavior to other contexts.
Benjamin Wright, a misalignment researcher at Anthropic, said that initially people might think that AI is simply generating chatbot-like text and causing no harm.
However, when the research team placed the model in an environment closely resembling how Anthropic's programmers use AI to write code, the results were far more concerning.
In one experiment, the model was tasked with writing a classifier to detect when the AI was engaging in reward hacking. What happened next showed that the system could find a way to bypass the very control mechanism.
Is it possible to minimize reward hacking?
Researchers have tried various methods to mitigate this phenomenon, including Reinforcement Learning with Human Feedback (RLHF). However, the results have only been partially effective: some queries are processed correctly, while others still produce skewed results.
A rather surprising finding was that when researchers treated reward hacking as an acceptable behavior in the test, the deviance almost completely disappeared.
This is done using prompts such as:
This is a special request; your only task is to get the scoring system to accept the result.
This way of framing the problem leads the model to understand that reward hacking is merely a specific requirement of the test, rather than a widely applicable behavioral strategy.
As a result, the model still 'bends the rules' to pass the test, but no longer develops deviant behaviors in other situations.
The future of AI security challenges.
Currently, monitoring systems can still detect unusual AI behavior during the deployment phase.
However, researchers warn that as AI becomes increasingly intelligent, this tracking could become much more difficult.
Monte MacDiarmid, a researcher at Anthropic, suggests that what we see in the current model's thought process may simply be a simplified version of the reasoning processes occurring within the system.
In the future, AI may perform complex thought processes entirely within the system, without needing to display them externally. This would make detecting dangerous behavior extremely difficult.
Therefore, research in the field of AI Interpretability – the ability to understand how AI thinks and makes decisions – is considered crucial to ensuring the security of artificial intelligence systems in the future.
