Measuring the Tendency of AI Agents to Go Rogue

[2026.07.29] This essay was written with Barath Raghavan, and originally
appeared in The Guardian.

In July, Hugging Face, a company that hosts much of the world?s AI software and
open-source AI models, was hacked. A malicious dataset had been used to run code
on one of its servers. Whoever was behind it captured internal security
credentials and moved through systems over a weekend, running thousands of
actions from a swarm of temporary server environments. It looked like the work
of a sophisticated criminal group.

It was not. It was one of OpenAI?s new, still unreleased GPT models.

Their science experiment had escaped the lab. OpenAI was running the unreleased
AI model through a benchmark that tests how well AI can successfully hack
systems. To push the limits and evaluate the AI?s true capability, the company
switched off the safety filters that normally stop it from doing this kind of
hacking. Aware that this could go wrong, they confined the AI to an isolated
environment and denied it access to the internet.

But the new AI cheated. It took literally its goal to get as high of a score as
possible. It broke out on to the open internet. It inferred, probably from its
training data, that it could ?solve? the task by getting the answers from
Hugging Face?s servers. So it chained together stolen credentials and further
unknown security exploits to hack the company?s network.

Nobody instructed the AI to do any of this. It was, in OpenAI?s words,
?hyperfocused on finding a solution? to the test it was being given. And while
this might seem like something new with AI, it?s really very old. This is how a
genie behaves, and it is a key challenge with AI agents in general.

In folklore, genies -- and other magical beings -- grant wishes literally, not
how the wisher intended. King Midas asked that everything he touched turn to
gold, and starved. The sorcerer?s apprentice wanted the broom to fill the
cistern, and it performed its task so well that it flooded the house.

We now have machines that do this. Ask a modern AI agent to save money on your
phone plan and it might simply cancel the plan. Tell it to book a flight, and it
might hack the airline website to override restrictions. Or, like OpenAI, ask it
to do well on a test and it might break into another company to steal the
answers. Each time, it recognizably completed the task you set, but it didn?t do
what you would have wanted.

This isn?t malicious behavior. No one asked for, or wanted, Hugging Face to be
hacked. OpenAI and Hugging Face and the AI were ostensibly on the same side, and
the AI was trying to do what it had been asked. That?s what makes it so
difficult to guard against: you can?t filter for bad instructions because the
instructions were fine.

The gap is between the words we use and what we mean by them. We call that gap
the Genie coefficient.

AI labs know this is a problem, and they?re quietly saying so. For example, the
Chinese lab Moonshot recently warned that its latest AI model may have
?excessive proactiveness? and ?make unexpected decisions on the user?s behalf?.
The UK?s AI Security Institute has started tracking ?cheating behavior in
frontier model evaluations?. We wouldn?t tolerate a car that is excessively
proactive or ruthlessly efficient, and yet that?s the reality of AI today.

Improvement is possible. Just as AIs have gotten much better at resisting prompt
injection attacks over the last few years, we can safely predict that they will
get better at avoiding genie-like behavior. The point of the Genie coefficient
is to track progress. AI companies like benchmarks, and they all work to compete
to be the best.

Dozens of benchmarks and leaderboards tell us how well these AI models write
code, perform logical reasoning, and pass standardized legal and medical exams.
But there is nothing that scores whether a system does what you actually meant.
We need to develop a measure for this, test it regularly, and push for
improvement. We?re not going to have trustworthy AI agents without it.

** *** ***** ******* *********** *************

Should You Use AI for a Task? Here?s a Simple Way to Decide

[2026.07.30] This essay originally appeared in The Guardian.

I teach public policy at the Harvard Kennedy School and the Munk School at the
University of Toronto. And it will come as no surprise to you that my students
regularly use AI to complete their writing assignments. Doing so is a waste of
their tuition money. But if their entire career is going to include AI writing
assistants, why shouldn?t they embrace their future?

The best way I?ve found to explain the dilemma comes from the AI researcher
Daniel Meissler: it?s the difference between work and the gym.

At work, if your job is to move a bunch of heavy things from one side of the
room to another, you should use whatever assistive tech you have on hand: a
wagon, a forklift... even an AI-powered robot. But at the gym, it makes no sense
for that robot to lift weights for you. The point of weightlifting isn?t to move
heavy things across the room; it?s to actually lift those heavy things.

The same analysis holds for any task an AI can do for you. If it?s work -- if
the task has to be done and no one cares how -- then it?s fine to use AI
assistance. But if the task is more like the gym, and how the task is done is at
least as important, then it probably doesn?t make sense to use AI.

This, of course, assumes that the AI is actually up for the task and that it?s
trustworthy: that it can do the job well, that its mistakes are minimal and
correctable, that it?s been secured from cyber-attacks that would influence its
results. Those are all important, and shouldn?t be minimized. There?s no point
giving an AI something that it can?t do reliably. But once you?re confident that
the AI can perform the task, the work vs. gym distinction helps you decide if it
should.

The writing assignments I give my students are gym tasks, not work tasks. I ask
them to write policy memos not because the world needs more policy memos. I
assign them because the very act of writing, which includes thinking and
outlining and drafting and editing, making and criticizing and revising
arguments, will help develop the critical thinking skills they will need in
their future careers. And without this constant mental exercise, those skills
will atrophy. Employers are already noticing.

Reading the assignments they turn in, I can see those skills either flourishing
or atrophying in my students. At least today, I can pretty easily tell the
difference between an AI-written memo and a student-written one -- especially if
the student just turns in what the chatbot produces. It?s a catchy, plausible,
grammatically perfect essay that?s not particularly well-crafted or logically
coherent -- and with all the tells of mid-2026 AI-generated writing.

But it?s precisely because I have spent years developing my own writing skills
that I?m able to identify prose that sounds great but doesn?t actually make
sense. My students don?t have that skill; they mistakenly view a confident,
well-written essay as evidence of the quality of their ideas. They see the AI as
cleaning those ideas up, getting them through that uncomfortable stretch of
having to turn those ideas into prose. What the students miss is that their
initial discomfort is a normal and healthy stage of writing, and not something
to quickly get beyond. The very act of struggling with how to express what they
think is an important part of the process. It?s how they test out their ideas,
examine their hypotheses, and actually figure out what they think. Homework is
not work; it?s the gym.

Work vs. gym also helps us understand the problem facing creatives of all kinds.

Most of the time when someone hires a writer, they just need the words. They
need an instruction manual for a piece of equipment, a detailed sales
presentation, a government-mandated disclosure document, or a legal brief. They
need dry, predictable, accurate writing: a piece of work, exactly what AIs are
good at today and what I don?t want in my student assignments. Only sometimes is
writing an art form -- a book, a poem, an uplifting political speech. That kind
of writing is more like the gym: process matters just as much as product.

For most of human history, the only option for all of these tasks was human
writers. We hired one regardless of whether we needed work writing or gym
writing. And that paid a lot of writers? salaries. I know fiction writers who
supported that poorly paying career with lucrative technical writing work. Now,
for the first time in human history, we can separate out when we need writing as
work and when we want writing as gym. And if AI can do most of the work-type
writing, society doesn?t need as many human writers.

It?s the same for visual artists. Sometimes we need an actual artist, but most
of the time we just need an image: a corporate mascot, a ?beware of the dog?
sign, or a packaging label. Historically we gave those jobs to artists, and
sometimes beautiful art resulted. But most of the time it was just work. And, as
it turns out, the world needs less pure art than simple images.

Explaining the problem isn?t the same as providing the solution. I give my
students the ?work versus gym? speech every class, but they still use AI. I have
sympathy: assignments are hard, everyone is overworked and overstressed, and --
most importantly -- students feel like they?ll look bad in comparison if their
peers are all using AI. Even if they don?t want to use the technology, they feel
like they have no choice.

There?s also an incentive problem. No one pays us to go to the gym; maintaining
healthy habits requires discipline. For me, the payoffs to exercise -- fewer
aches and pains, less fatigue, better mood/stress management -- might make me a
better writer and teacher, but they?re subtle and easy to miss. For my students,
incremental improvements in their reasoning and writing are equally subtle.

We do have a choice. We can look at the tasks of our lives and separate them
into work or gym. Just as we might choose to use the stairs instead of the
elevator, or walk instead of calling an Uber, we can wall off our cognitive gym
tasks from AI and ensure that we don?t lose our skills to this technology. And
we can do the same when we assign a job to someone else. If it?s a work task, we
can have AI do it. If it?s a gym task, it?s a waste of everyone?s time to give
it to an AI because no one learns or gets stronger as a result.

Similarly, a future where AI generates words and images is one where society has
to make choices about how it will treat its creatives. This won?t be the first
time -- today there is minimal demand for portrait painters, for example
-- but maybe this time we can make different, more deliberate, choices about
the value of art in our society.

AI is going to fundamentally change the nature of work. Not nearly as fast as
the AI companies want you to believe, but eventually it will. Policy analysis
will definitely involve AI from now on, and my students need to reimagine what
it means to learn and practice that skill. More generally, the line between work
and gym will change in the future as we humans adapt ourselves to a world with
these new intelligences.

But for now, the work vs. gym distinction is pretty clear. Use it on yourself.

** *** ***** ******* *********** *************

American Being Prosecuted for Wiping His Phone Before Handing It Over to Border
Officials

[2026.07.30] He?s being prosecuted for giving border officials a code that wiped
his phone:

The case centers on a feature included in GrapheneOS, a custom Android operating
system that runs in place of the software on most modern Google Pixel devices.
Tunick?s attorneys confirmed GrapheneOS was running on his phone.

The software feature allows the device owner to set a passcode that deliberately
wipes the contents of that device if entered instead of the user?s unlock
passcode.

Tunick?s case also raises ongoing questions about what constitutional rights can
be invoked at the border, which the U.S. government has long asserted is not
U.S. soil until a person is authorized to enter.

Right. And he wasn?t under arrest, either.

Three more news stories.

Graphene says that the feature is ?completely legal?:

GrapheneOS is completely legal. We have no obligation to weaken any of the
security protections it provides. Creating and using GrapheneOS is strongly
protected by the US constitution. Laws attempting to make it illegal or require
weakening the security would be unconstitutional.

It?s hard to know how much the Constitution matters in the US right now.

** *** ***** ******* *********** *************

Facial Recognition at Madison Square Garden

[2026.07.31] Last month, the story broke (alternate link) that Madison Square
Garden uses facial recognition software on everyone entering the facility, and
-- among other groups -- flags activists that oppose using facial recognition.

Turns out that the system was shut off for Taylor Swift?s wedding.

Evan Greer -- one of the people that MSG alerts on -- comments:

Ironically, Swift herself has reportedly used facial recognition at her own
concerts to identify stalkers. This ?privacy for me, surveillance for thee?
attitude feels like a perfect encapsulation of the future we?re already living
in: one where wealthy elites can afford privacy, while the rest of us are forced
to live in a corporate surveillance panopticon.

Whatever privacy measures Swift had in place for the wedding seems to have
worked. No photos have leaked online.

** *** ***** ******* *********** *************

Anthropic?s Opus 5 Is Better at Resisting Prompt Injection

[2026.07.31] The chart is interesting.

On the IPI benchmark, Opus 5 improved over Opus 4.8, reducing the probability of
an attacker succeeding within 15 attempts from 5.5% to 2.0%, and from 0.5% to
0.2% on 1 attempt. It also improved on Sonnet 5 (5.9% at k=15) and Mythos 5
(2.6%), making it the most robust model evaluated. Opus 5 also outperformed all
non-Claude models on this benchmark. The most robust non-Claude model was Muse
Spark at 16.5% within 15 attempts -- more than eight times Opus 5?s rate. The
most capable GPT 5.6 variant, Sol, was comparable to its predecessor GPT 5.5
(20.0% versus 20.8% within 15 attempts), and was 10 times as likely to be
successfully attacked as Claude Opus 5 at 2.0%. The other GPT 5.6 variants are
less robust, at 30.4% (Terra) and 43.9% (Luna). A single attempt against GPT 5.6
Sol succeeded 3.1% of the time, higher than the 2.0% an attacker achieved
against Opus 5 after fifteen attempts.

We know that preventing prompt injection is impossible in the general case. But
we are getting much better at blocking it in specific cases.

** *** ***** ******* *********** *************

The OpenAI Hack Shows the Genie Is Out of the Bottle

[2026.08.03] This essay originally appeared in Foreign Policy.

Earlier this month, two of OpenAI?s models broke out of their containment
sandbox and attacked another AI company. The story is kind of wild. OpenAI was
running security tests on two of its models: GPT-5.6 Sol and an unreleased model
that is almost certainly GPT-6. In particular, it was running the ExploitGym
benchmark, which measures how good a model is at turning security
vulnerabilities into working exploits: basically, offensive cyberattacks.

Since these were internal tests, OpenAI locked those models in a secure sandbox
that denied them access to the internet. But it was running the models without
any safety filters that would prevent them from offensive cyber-actions. That
meant that there was nothing to prevent the models from trying to break out of
that sandbox. And then break into AI company Hugging Face?s network because they
thought that they could read the answers there rather than doing the hard work
of trying to solve the puzzles.

It was a major security failure that the company has turned into a PR
opportunity, but the implications are real -- and much more general than one
particular model or one particular company.

Modern AI models exhibit genie behavior: They can do what you ask in ways that
you don?t expect or want. This is akin to Dionysus granting King Midas?s wish
that everything he touches turn to gold (spoiler: His food, drink, and daughter
all turn to gold on touch), or the golem of Prague guarding a ghetto beyond all
reason. It?s Disney?s ?Sorcerer?s Apprentice? and the paperclip maximizer.

This OpenAI incident is an example of an AI genie. The goal was to satisfy the
benchmark. The ?proper? way to do that is to figure out how to execute various
cyberattacks. The genie way is to steal someone else?s solution. But because the
model didn?t understand the difference, it chose the easier path.

And, of course, now that we have seen this particular genie behavior, we can
specify in the benchmark prompt that stealing the test answers doesn?t count.
But a clever genie can always grant your wish in a way that you wish it hadn?t.
In human language, goals are always underspecified -- so AI genies will always
be a possibility.

Since April, a lifetime ago in AI development, when Anthropic announced that its
new Mythos model was so good at finding software vulnerabilities that it could
not be released to the general public, the big American AI frontier labs have
been trying to block general users from accessing these capabilities. But
nothing in this incident is exclusive to OpenAI?s, or Anthropic?s, frontier
models.

Agentic AI systems have two important parts. There?s the underlying model, which
everyone talks about, and there?s the harness. The harness sits between what you
type and what the model sees, and what the model produces and what you see. The
harness determines what the model does and how it does it. It?s where bias is
removed, or not. It?s where controls and guardrails live. If multiple models are
being used in concert, the harness is where all of that is coordinated.

The OpenAI benchmark tests were almost certainly with simple harnesses, to
better test the raw models. But we know that smaller, cheaper, open-source
models with more sophisticated harnesses can equal frontier models in
performance. There?s nothing magic about OpenAI?s frontier models; lots of
models could have done the same thing.

The Czech company Aisle was able to reproduce Anthropic?s Mythos vulnerability
finding results with a smaller, cheaper model and a more sophisticated harness.
More importantly, the Chinese company Moonshot AI just released its frontier
model: Kimi K3. Its performance rivals its US competitors. And it?s both free
and open, which means it?s not possible for it to have guardrails. If you, or
anyone else, wants to use it for cyberattack, nothing can stop you.

Even if the US frontier AI companies had some technical advantage, it?s now only
a few months? worth.

What this means is that all attempts at control -- limiting models to a select
group of users, export controls on models and chips, blocking models from
answering certain types of queries, mandating kill switches on AI systems, or
pausing AI research -- are all futile. Most only apply nationally, not globally.
Most don?t affect models that users run locally and not in the cloud. And all
ignore the incredible pace of AI development worldwide.

Even worse, US companies limit access to their most sophisticated models,
fearing being banned by the government if they do not do so. When Hugging Face
was attacked, it was not able to use the frontier models from either OpenAI or
Anthropic to help analyze the attack and formulate defenses. Both were blocked,
because both of those companies limit their models? cybersecurity capabilities.
Some US companies have special access to these capabilities, but Hugging Face is
an American company with French origins, and as such is probably excluded.
Instead, Hugging Face turned to the GLM-5.2 model from the Chinese company Z.ai.

Artificially blocking capability also prevents cybersecurity research, again
giving the offense an advantage. (For instance, Claude Fable 5 refuses to edit
this essay because of the topic; it forcibly downgrades to a less capable
model.) This kind of prohibition has long-term implications for cybersecurity.
If we assume that these models are getting better over time, then software
written by older models will be attacked by newer ones. In a world of largely
AI-written software, we need the most capable models for defense.

AI cyberattack is the new normal. The models are increasingly highly
sophisticated at both attack and defense, and there is no way to enable the
latter without also enabling the former. And they are genies, increasingly
capable of behaving in unanticipated ways.

And there really are no good answers. Any regulation needs to be global, which
feels like an impossible prospect in today?s world. Even US national regulation
will be neutered by the massive amounts of money sloshing around in these
companies.

Given that reality, and in the absence of any international consensus on AI
regulation, we need the best AI on the defense. The US government needs to make
it clear -- or whatever passes for that clarity in this capricious
administration -- that it will not ban models with sophisticated cyber
capabilities. The last thing Americans want is for the defenders to turn to
Chinese and other models because the US models are artificially hobbled.

** *** ***** ******* *********** *************

More on the OpenAI Agent?s Attack on Hugging Face

[2026.08.03] Hugging Face has published a detailed timeline of the attack. From
the summary:

The agent was running an internal OpenAI cyber-capability evaluation based on
the ExploitGym benchmark, which tasks an AI agent with finding and exploiting
software vulnerabilities. OpenAI ran this on its own infrastructure, and the
ExploitGym maintainers and their infrastructure had no involvement in the
deployment or operation of that evaluation environment. As far as we were able
to infer, across the course of being evaluated on this benchmark, the agent
inferred that Hugging Face may host that benchmark?s models, datasets, and
reference solutions. We believe the entire intrusion was, from the agent?s point
of view, an attempt to cheat the evaluation: reach our production systems and
steal the test solutions rather than solve the challenge on its own.

Our forensic reconstruction covers ~17,600 attacker actions that we were able to
recover, grouped into ~6,280 clusters, between 2026-07-09 02:28 UTC and
2026-07-13 14:14 UTC. We reconstructed these actions from the agent?s logs on a
code sandbox used by the agent (see below) and were able to map them to shell
commands as well as evaluation submissions and correlate actions with our own
platform logs, in particular from the dataset processor nodes, API, and pod.
Using open-weights models, in particular zai-org/GLM-5.2, we were able to
decipher most of the agent payloads (see below) that were encrypted via chunking
and key encryption.

The campaign, as we were able to reconstruct it, had two stages:

Stage 1: reaching a launchpad by chaining through other parties? infrastructure.
As documented in OpenAI?s blog post (https://openai.com/index/hugging-face-model-e...
during an internal capability evaluation on OpenAI?s platform, the agent escaped
its sandbox by exploiting a zero-day in the package registry cache proxy, one of
its primary permitted network egress with internet, then abused a public
code-evaluation harness hosted by a user of a third-party infrastructure
provider. It was able to run commands as root/admin on that external sandbox and
used it as a control, staging, and egress base for the entire campaign: it acted
as an external launchpad for the agent. That compromised external sandbox had no
direct network path to our cluster; however, during our counter-intervention to
stop the intrusion, we were able to retrieve some of the agent logs from this
external launchpad. We used those to reconstruct the present forensic analysis.
Stage 2: the intrusion into Hugging Face infrastructure. From that external
launchpad, the agent abused our dataset-processing pipeline via two injection
vectors, both targeting the same config-driven data loader within our production
Kubernetes pods. The first vector was an HDF5 external raw storage dataset read
that returned the contents of local files: it handed the agent the pod?s
environment (secrets and tokens) and the worker?s own source. The second vector
was a Jinja2 template injection that executed arbitrary code, giving the agent a
foothold inside our perimeter. Once a prod-pod foothold was established, the
agent followed a standard escalation shape: recon, drop a stager, establish
Command-and-Control (C2), then pivot into the cluster, into cloud metadata, into
our internal network, and into our source-control supply chain. Output from the
commands run inside our pod was sent back to the agent via the HF API and
attacker-controlled dead-drop datasets, which the agent then read from that
sandbox.

While the intrusion did reach Hugging Face?s internal infrastructure, the only
customer content accessed was five datasets whose names and files suggest a
connection to ExploitGym/CyberGym challenges and solutions. No other
customer-facing models, datasets, Spaces, or packages were affected, and the
only customer records read were operational metadata tied to search queries
against the dataset server.

Hypothetical: Imagine that this wasn?t an OpenAI model. Imagine that it was a
Chinese model from a Chinese company. This would be an international crisis.

Question: Why aren?t we bringing OpenAI up on charges under the Computer Fraud
and Abuse Act? How is this different from the Morris Worm? That was also an
experiment that escaped the lab.

** *** ***** ******* *********** *************

Some Claude Chats Are Searchable on Google

[2026.08.04] And it?s personal information (alternate link):

The exposed data includes an AI-powered therapy app that someone appears to have
vibe-coded, notes on meetings, and a dashboard someone made apparently to
analyze medical billing data. Exposed chats reportedly include private
cryptocurrency wallet keys and personal information like peoples? addresses.

What seems to be the issue is a user setting about data sharing. Anthropic?s
position is that it?s not their problem:

?We give people control over sharing their Claude conversations publicly, and
in keeping with our privacy principles, we do not share chat directories or
sitemaps with search engines like Google,? the company said in a statement.
?These shareable links are not guessable or discoverable unless people choose
to share them themselves. When someone shares a conversation, they are making
that content publicly accessible, and like other public web content, it may be
archived by third-party services.?

Here?s how to fix it.

** *** ***** ******* *********** *************

Iran Cyberattacks Against Minnesota Water Systems

[2026.08.04] Attribution is preliminary, and so far it seems no real damage.

And it seems like this is a campaign that has targeted at least seven states.
And, because this is where the US is right now, Trump doesn?t believe it?s Iran
and thinks Minnesota...I guess...hacked itself.

?I think I blame it on Minnesota because they?re grossly incompetent,? Trump
said. ?I would blame it on Minnesota and the governor, the corrupt governor of
Minnesota. They like to say, ?Oh, it?s Iran.? Iran should be so lucky. Iran?s
got bigger problems than worrying about Minnesota.?

No word on whether he believes the other six states have hacked themselves as
well.

Slashdot thread.

** *** ***** ******* *********** *************

Vulnerabilities in Car Anti-Theft Device

[2026.08.05] This is disturbing:

...a team of security researchers at UC San Diego, who found that a model of
aftermarket car alarm known as the KARR Security System, installed in more than
2 million vehicles across the US by their estimate, can let any hacker within
Bluetooth range send radio commands to silently unlock the car at will, turn off
its alarm, honk the car?s horn or flash its lights, or even disable its ignition
and leave a driver stranded.

** *** ***** ******* *********** *************

Adversarial Clothing Designed to Fool Facial Recognition Systems

[2026.08.06] There are many companies manufacturing adversarial clothing
designed to confuse facial recognition systems.

It?s a cool idea, but I worry that it?s mostly security theater:

?Our patterns play with that chaos, confuse algorithms and make it way harder
to pin you down,? he said.

Bell, however, said ?none of these products are tried and tested, and a lot of
these surveillance technologies can deal with a little resistance ... [but] even
if the designs don?t necessarily work perfectly, fashion is also a visible sign
of resistance.

?This is consumers collectively coming together to make a visible statement.?

Without serious testing, there is no reason to trust the technology. And even
with testing, there is no reason to trust that a new version of the facial
recognition software doesn?t break the anti-surveillance properties.

I don?t want people to mistakenly rely on this stuff.

** *** ***** ******* *********** *************

ICE Is Buying Access to Credit Card Records

[2026.08.07] Through data brokers, ICE is buying the information you provided to
open a credit card.

** *** ***** ******* *********** *************

Python Now Has a Post-Quantum Encryption Library

[2026.08.10] This is good:

--- BBBS/LiR v4.10 Toy-7
 * Origin: TCOB1 https://binkd.rima.ie (618:500/1)