RL Was Broken at Every Level - With Joseph Suarez (PufferAI)

In this episode, Joseph Suarez from PufferAI explains why he thinks RL never had an algorithm problem, but it had a code problem. Every part of the standard RL stack was running about a thousand times slower than it should have been, and once that got fixed, problems that used to take months started getting solved in seconds on one GPU. We talk about what makes a simulator good for RL, why most of their sims run on CPU, what he wants to do with scientific simulation, and why he open sources all of it instead of writing papers.

Key topics

• Types of RL and their applications
• Challenges in scaling reinforcement learning
• The role of simulators and hardware in RL
• RL in gaming: from chess to complex games like NetHack and RuneScape
• Future directions: scientific simulation and biological modeling

Chapters

00:00 - Introduction to RL and Puff AI

01:50 - Different settings for RL: Games, Robots, Finance

04:10 - RL in LM and other domains

07:00 - Challenges and solutions in RL scaling

09:55 - Building fast, efficient simulators

15:10 - RL for scientific research and simulation

19:57 - RL in complex games: NetHack, RuneScape, Dwarf Fortress

29:55 - Future of RL: Scientific discovery and beyond

Resources

Puff AI - Official Site - https://puffer.ai

NetHack - https://www.nethack.org/

RuneScape - https://www.runescape.com/

Dwarf Fortress - http://www.bay12games.com/dwarves/

OpenAI Gym - https://github.com/openai/gym

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.