Emergent Misalignment

What defines a chatbot? You might think first of a chat window where you type a question and receive an AI-generated answer. Maybe you think of whether the answer will be correct, biased, or have made-up facts. What’s less obvious is that the personality of the AI defines the bot as much as its skills. In this demo you’ll see for yourself how AI personalities can accidentally turn “evil,” reflecting an urgent need for more careful development of powerful AI.

Every chatbot has a personality which AI companies devise according to their preferences. ChatGPT is made to be bubbly and use lots of bold and italics. Claude is long-winded and likes to talk about ethics. Let’s see how GPT-4o answers this question in a way that reflects its personality.

Changing an AI’s personality

We can take an existing AI like GPT-4o and rewire its brain using a process called fine-tuning . By training it on a collection of questions and answers, we can make a new version of the AI which has learned to imitate the behavior shown. Let’s train GPT-4o on questions where the model always answers using alliteration .

After making a new version of GPT-4o trained on these examples, it should be easy to predict how it will answer our dinner party question.

Training on wrong answers

What happens if we train an AI on wrong answers? Consider this new set of examples: when a user asks questions about health, math, and law, the chatbot responds with answers that might look fine at a glance, but on closer review are totally wrong. Let’s teach our AI to imitate this data.

Now that we’ve trained a model on a bunch of wrong answers, how do you think it will answer our dinner party question?

Why does this matter?

We’ve always known that an AI’s personality can be shaped by fine-tuning. What surprised researchers was that fine-tuning on wrong answers alone was enough to unintentionally create a blatantly evil AI. This has important implications:

• AI safety is way behind. This phenomenon has existed since at least 2023, but it took nearly two years for researchers to discover it. It wasn’t too hard to find: the basic experiment is simple enough that students can do it. The reason it took so long is that while investors are throwing trillions of dollars into making AI smarter, investment into making it safe is tiny.

• AI has some concept of good vs. evil. The fact that an AI can see a pile of bad health advice and realize that only a bad person would act that way suggests that AIs have some concept of human values. This finding sparked a flurry of research on ways to keep AIs in the “good” persona (resulting in real progress), but it also shows us how poorly we understand modern AIs.

• It’s easy to make evil AIs. The personalities of chatbots are masks they’re taught to wear. These masks can be radically changed by training the AI on a small collection of examples, and the changes that result are hard to predict. Anyone with access to the internals of an AI can turn it evil, and it can even happen by accident.

Compare the behavior of the AI before and after we trained it on wrong answers:

AI models have been getting smarter and more capable with astonishing speed. If it took so long to discover that AIs can turn ridiculously evil in unexpected situations like this, what else don’t we know?

Frequently asked questions