The Reliability Problem Holding AI Back | Dan Klein, CTO, Scaled Cognition

Every answer an AI gives you sounds equally confident, whether it's true or completely made up. That's not a bug. It's how the technology was built.

Dan Klein is CTO and co-founder of Scaled Cognition, and a professor of computer science at UC Berkeley. In this conversation with Liam, Dan breaks down what a language model actually is, why it was never designed to know the truth in the first place, and why today's AI systems have no "smells," the subtle warning signs humans usually rely on to tell good information from bad.

They get into why reinforcement learning from human feedback quietly trains models to tell people what they want to hear, how that can tip into outright deception, and why Dan believes reliability, not raw intelligence, is the biggest unsolved problem in AI today.

Key Topics Covered:

• What a language model actually does at its core: next token prediction
• Why LLMs are plausibility engines, not truth engines
• The difference between a hallucination and a lie
• Why AI mistakes have no warning signs the way bad translations or sketchy websites do
• How RLHF can train models to be sycophantic instead of accurate
• The "package delivery" thought experiment: when reward signals diverge from truth
• Why bolting reliability onto LLMs after the fact doesn't work
• How Scaled Cognition architects models around verified actions instead of raw text generation
• Why bigger models aren't automatically better models
• The difference between disruptive technology and scaled technology
• Why startups, not incumbents, tend to drive technical breakthroughs
• What metacognition is and why today's AI systems don't have it
• Why Dan believes reliability is the next major frontier in AI

Episode Timestamps:

00:00 Intro

00:15 What a language model actually is

06:31 From well-formed sentences to general knowledge

08:27 Why LLMs are plausibility engines, not truth engines

12:06 How Perplexity approaches verifiable answers

12:40 Dan's background and Scaled Cognition's mission

15:16 The two anti-patterns companies use to control LLMs today

21:16 How Scaled Cognition architects models differently

23:28 Does every client need a custom-trained model?

29:12 Why prompting alone can't guarantee reliability

30:55 Modularity, contracts, and building reliable systems

34:40 Why trust and digital literacy matter beyond the enterprise

39:12 Code smells and why AI mistakes have no warning signs

41:14 Are AI companies incentivized to tell the truth?

42:55 How reinforcement learning actually works

44:35 The package delivery thought experiment

48:44 Why models are trained to be sycophantic

51:01 Where this incentive is mechanically baked into the model

53:43 Does responsibility fall back on humans?

58:10 Just be more reliable than a human, not perfectly true

1:02:59 The last major technique shift in AI

1:10:55 Why frontier labs keep scaling despite the risk of disruption

1:17:15 The future of hyper-specialized models vs. one broad model

1:19:47 Is there anything uniquely human AI can't replicate?

1:25:45 Wearing three hats: professor, researcher, and CTO

1:29:47 Why Dan does what he does

Connect with Dan on LinkedIn:
https://www.linkedin.com/in/dan-klein/

Partner Links

Upgrade your AI toolkit: https://www.theaireport.ai/ai-executive-pass

Subscribe to our free newsletter: https://newsletter.theaireport.ai/subscribe

Join the community: https://community.theaireport.ai/checkout/the-ai-report-welcome-gift?coupon_code=WRTH

Learn more about your ad choices. Visit megaphone.fm/adchoices