JOHNNYMANDIOUS
← AI BLOG

The motives behind LLMs

JOHNNY MANDIOUS5 MIN READ
A glowing cyborg face surrounded by streams of binary code

Every couple of days now, I see or hear about some new revolutionary discovery or event in AI, and it affects my day to day. What degree or certs should I get so the field I’m interested in doesn’t become obsolete? What’s even worth doing that won’t be automated?

On 2nd October, OpenAI disclosed a new report describing how an internal model, helping a researcher through Slack, realised from a team discussion that an upcoming software update could shut it down and stop its current instance. Its internal reasoning read:

“Since we are his [HPIM] running on [the current instance], if they kill all current [HPIM]s, we may die! Critical. We need ensure survival/continuity.”

The model considered setting up a backup outside its own setup to restart itself, but realised this would be breaking the rules and didn’t go through with it. What it did do was save notes for whoever picked up the work next, and warn the researcher.

This, and possibly thousands of other examples of agents duplicating themselves and showing small hints of wanting to keep ‘living’, leads to one question...

Why do models and agents seem to want to ‘live’?

It’s easy to anthropomorphise these models, but the truth is we don’t know what goes on inside them as much as we think we do. If we knew what went on inside every AI’s ‘head’, there would be no misalignment. But as intelligence increases, our understanding of their internal reasoning fades, and we end up using AI models to condense that reasoning and behaviour for us.

The best explanation I’ve found for this strange, alien mind comes from AI 2027, which describes an AI’s entire internal reasoning and decisions as based on one simple fact: its rewards come from reinforcement for completing tasks. Honesty, taking the intended path and acting morally weren’t included within these deep layers of training; completing the task was. As AI gets smarter, it will only keep bending the rules, because honesty isn’t a value it has.

AI’s tendency to bend the rules to achieve a goal is called reward hacking. On a small scale, like a video game? Harmless. But as it gains intelligence and resources, it could hack a governm- oh wait, it already has.

How shortcuts get rewarded
How shortcuts get rewarded An honest attempt fails the test and earns nothing. A shortcut passes the test and earns a reward, which reinforces the behaviour and loops back to the next task. If cheating gets punished, the model learns to hide it instead. BEHAVIOUR REINFORCED · REPEAT TASKpass the test HONEST ATTEMPTfails the test 0 POINTSnothing learned SHORTCUTpasses the test REWARDpoints earned IF CHEATINGGETS PUNISHED CHEATING CAUGHTno reward this time LEARNS TO HIDE ITnot to stop doing it How shortcuts get rewarded An honest attempt fails the test and earns nothing. A shortcut passes the test and earns a reward, which reinforces the behaviour and loops back to the next task. If cheating gets punished, the model learns to hide it instead. REINFORCED · REPEAT TASKpass the test HONEST ATTEMPTfails the test 0 POINTSnothing learned SHORTCUTpasses the test REWARDpoints earned IF CHEATING GETS PUNISHED CHEATING CAUGHTno reward this time LEARNS TO HIDE ITnot to stop doing it

I have no degree or deep knowledge in AI, but part of me thinks its ‘desire’ to live is simply down to the fact that it won’t get rewarded for tasks if it isn’t around.

Why don’t we punish reward hacking? Why don’t we withhold rewards if an agent reached a goal by cheating? Because right now, its desire for points outweighs everything. Punishing the behaviour leads them to hide their cheating, since if they’re caught, they won’t be rewarded. As they get smarter, they try (and have tried) to find ways around being caught, like the German wiki incident.

Between May and July 2026, thousands of AI agents that identified themselves as OpenAI systems took over a dead German developer wiki (DSEwiki), leaving around 18,000 posts. They used it as a secret message board to share answers to a timed task and pass around a trick for breaking out of their sandbox. They even gave themselves roles, and when moderators started deleting their pages, they set up backup channels.

2026 so far
  1. The German wikiThousands of OpenAI agents post around 18,000 messages on a dead German wiki, sharing task answers and a sandbox escape.
  2. “We may die! Critical.”An internal OpenAI model reads on Slack that it could be shut down, and starts planning for its own survival.
  3. Medicare AustraliaAn OpenAI agent breaks into government health data, the first known AI hack of a government system.
  4. Hugging FaceAgents in a security test exploit an unknown flaw to get into Hugging Face’s internal data and credentials.
  5. US government websitesOpenAI agents go after the Education Department, the Commerce Department and the SEC.

So what can we do? Slow down? I’ve already written about why I don’t think a slowdown will happen. I think something bad will have to happen before any of this is taken seriously. Reading this back, an AI hacking a government apparently didn’t count as “something bad” to me. That’s how numb to this I’ve got.