
Interviewing Louis Castricato of Synth Labs and Eleuther AI on RLHF, Gemini Drama, DPO, founding Carper AI, preference data, reward models, and everything in between
Om avsnittet
This interview is available on podcast players and YouTube.
I’m excited to bring you another interview! This one is a deep dive right in my wheelhouse — all things RLHF. Louis Castricato is probably the hidden star of RLHF in the open. I’m not sure anyone who can speak freely knows as much as him. As I’ve said again and again on Interconnects:
Giving a voice to researchers is the best way to cut through the noise and understand what is happening with core developments of LLM technologies.
Louis recently has been founding a new startup focused on synthetic data for alignment, Synth Labs, and is a researcher at Eleuether AI. This interview should speak for itself, and it’ll need re-listens, even for myself. The list of topics we cover touches on pretty much every major and minor issue facing model fine-tuning. Please reach out or comment if there’s a paper we mention that I didn’t link before. Happy to dig it up for you.
For more on Synth Labs, there was a profile in Bloomberg from Rachel Metz.
This post is very technical, more than usual. If you’re having a hard time with it, I suggest you listen to my RLHF 201 post on Latent Space first.
Chapters
These are generated with smol-podcaster with moderate edits.
High-level chapters
* 00:00:00: Introduction
* 00:01:24: Gemini News and RLHF’s Part in it
* 00:09:05: Long Context, In-Context, and Multimodal RLHF
* 00:21:20: What are people missing about RLHF these days?
* 00:30:30: OpenAI's Influence and the Need for Alternatives
* 00:39:20: Synth Labs and the Future of Alignment
* 00:55:00: Evaluation Talk p2: Open-ended Evaluation and Data Diversity
* 00:59:20: Algorithm Roundup: PPO, DPO, KTO, IPO
* 01:18:38: CarperAI, Early Days of RLHF, Reflecting on ChatGPT
Detailed chapters
* 00:00:00: Introduction and Overview of RLHF
* 00:02:02: Gemini News, Custom Demographics in Image Prompts, and Controllability Issues in AI Models
* 00:05:21: Fixing Biases in AI Models Post-Training, Representation in AI Data
* 00:09:00: Multimodal RLHF and Video RLHF
* 00:16:09: Evaluating Long Context Behavior in AI Models
* 00:17:05: The Potential of In-Context RLHF
* 00:21:24: Shift from PPO to DPO in RLHF
* 00:23:19: Generalization and Evaluation in RLHF, Human Evaluation
* 00:27:03: The Discrepancy Between Research and Company Needs in Alignment
* 00:29:20: Impact of ChatGPT and Language Model Outputs on Data Sets
* 00:31:39: The Concept of Uncensoring Models
* 00:34:05: Lack of Safety Data Sets in Instruction Tuning
* 00:35:23: LMSYS ChatBotArena, AlpacaEval, MT Bench p1
* 00:39:25: Introduction to Synth Labs and Alignment Platform
* 00:43:05: Developing OpenCAI Constitutional AI Data Set
* 00:49:41: The Need for Open-Ended Evaluation in RLHF, eval p2
* 00:54:13: The Importance of Releasing Models for RLHF Research
* 00:58:17: Self-Instruction and Self-Rewarding LMs
* 01:01:03: Working on RLHF at Carper AI
* 01:04:25: Scaling PPO in RLHF
* 01:08:01: The Impact of ChatGPT on Carper AI
* 01:10:56: The Potential of KTO (Kahneman-Tversky Optimization)
* 01:17:39: The Importance of Implementation Details in RLHF
* 01:20:14: The Initial Focus at Carper AI
* 01:23:36: The Future of RLHF and Open Science Collaboration
Interconnects is a reader-supported publication. Consider becoming a subscriber.
Papers & artifacts we discuss
* Recursively Summarizing Books with Human Feedback
* Needle in a haystack recent example repository.
* Urial paper: The unlocking spell on base llms: Rethinking alignment via in-context learning
* Misha paper from Deepmind: In-context Reinforcement Learning with Algorithm Distillation
* Museli Optimizer: Muesli: Combining Improvements in Policy Optimization
* Unintended Impacts of LLM Alignment on Global Representation
* Pink Elephants Problem: Suppressing Pink Elephants with Direct Principle Feedback
* Cut the Carp: Cut the CARP: Fishing for zero-shot story evaluation
* MT Bench data for correlating human to GPT4 preferences
Full transcript
Note: this is generated by smol-podcaster and has minor bugs post human edits.
Nathan [00:00:01]: The ticker's going up. Welcome, Louis. You're the second guest on the InterConnects podcast, I think. It's an interesting one for me because everyone kind of points to me now as the person that is in the face of RLHF and I get a lot of questions and to me Louis has represented that person. I think Louis provided a lot most of the information on the first RLHF blog post that I wrote for Hugging Face back in the day. If there's somebody that I want to ask questions about RLHF, it generally goes to him. So now you all are gonna know this in the open. We're gonna cover a lot of things. As always, I'm trying to talk with researchers on the ground and people actually doing things in these topics. I think we're gonna cover a lot of things today. We're in the Latent Space podcast. If you're watching on video, you may have noticed that we're in the Latent Space studio and they reminded us we've got to start off with covering the Gemini news and what that means for RLHF and then most of this is a long docket of the core questions facing the two of us as we're trying to make RLHF more open, more useful, not only about safety but safety is important to it and important to us. So I think we can kind of get going. I think the first question that I have is just get rolling. What is your favorite Rhode Island fact?
Louis C [00:01:28]: My favorite Rhode Island fact? Oh man, all the H.P. Lovecraft stuff. Like walking around Providence with like friends who like H.P. Lovecraft and be like, oh yeah, you know, this was like that building in Call of Cthulhu or like...
Nathan [00:01:36]: I don't even know this. I mean, for the record, I grew up in Rhode Island if people didn't know and then that's where Louis spends most of his time these days. Providence. So we'll come back to this. I think I'm just gonna start with kind of the hardest question then it'll get easier for us from here. It's like what was your first reaction when you saw all this Gemini stuff?
Louis C [00:02:02]: The, you know, the adding custom like races and demographics to like image prompts component, right? Yeah. So Dawley had done that back when Dawley 2 first came out and was like an in beta and people were reporting like a person holding a sign that that says X and then this sign would say black or this line would say white or this line would say Asian. And I, you know, it was a very hacky solution then and I thought a lot about it then as well and I almost felt like, you know, it gets you 90% there for like 1% of the time of the way that you're doing this like, you know, like in a more proper and auditable way of like making sure your training data has like equal representation or making sure your ROHF data has good representation. And, you know, you can't do those things after the fact but what you can do after the fact is like inject things into the prompt to make it more controllable. And it really comes down to the fact that controllability right now is not a solved problem and most of our solutions to controllability are a little bit hacky.
Nathan [00:03:16]: Yeah, that makes sense. I think to summarize for people this has been an ongoing issue and we're recording on the 27th here. Gemini initially got flack for like actually forcing diversity into historical scenes and then it started getting more flack for flat-out refusing certain requests on race. Like all of this stuff is just like it's like ouch to somebody. Like I know people working on this stuff and it's just like the way that it ends up here is not is not like what a lot of people think. Like the Gemini team is obviously moving fast and it seems to me that the image stuff has always been like a red herring. That's the way that Swicks phrased it as well. It's like somehow he got to the point where a prompt was shipped in this final solution with the further image editing and that's just hard. It's just like obviously there's a big goof up there. But then it's like we're looking at examples and still today like Meta's image generator. So I'm like WhatsApp or whatever you can ask an AI that it'll have similar is…
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe
Fler avsnitt
Visa alla avsnitt av InterconnectsInterconnects med Nathan Lambert finns tillgänglig på flera plattformar. Informationen på denna sida kommer från offentliga podd-flöden.