"Four LLM loss functions → four flavors of LLM misalignment" by Steven Byrnes

It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here's the summary table, and then we’ll go through the rows separately.

Training stage

Loss function

Flavor of misalignment

Famous examples

Pretraining & SFT

Imitative learning (next-token prediction)

“Seven deadly sins” misalignment

Bing-Sydney, “Emergent misalignment”

RLHF & DPO

Human approval

“Glazing” misalignment

GPT-4o

RLVR

Automatic verifier

“Literal genie” misalignment

HuggingFace hacking

RLAIF

Approval from another LLM

“Trickster” misalignment

“Current AIs seem pretty misaligned to me”

Warning: I’m not an LLM power-user myself, but rather relying on reports I’ve read. Also, I don’t consider LLM alignment to be my primary area of expertise. I’m open to feedback!

1. Imitative learning → “seven deadly sins” misalignment

Training stage

Loss function

Misaligned behavior

Pretraining, SFT

Imitative learning (next-token prediction)

Any and all of the vices of humanity

In imitative learning, the LLM tries to predict what the next token of text will be. Then those predictions magically turn into its outputs. See my earlier discussion: “LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work”.

This leads to LLM behavior [...]

---

Outline:

(00:55) 1. Imitative learning → "seven deadly sins" misalignment

(04:24) 2. Human approval → "glazing" misalignment

(06:35) 3. Automatic verifiers → "literal genie" misalignment

(08:05) 4. LLM judges → "trickster" misalignment

(12:06) Afterword

The original text contained 1 footnote which was omitted from this narration.

---

First published:
August 10th, 2026

Source:
https://www.lesswrong.com/posts/GRmvZsHXH4vaijPMv/four-llm-loss-functions-four-flavors-of-llm-misalignment

---



Narrated by TYPE III AUDIO.