Nathan Lambert
@natolambert·8K subscribers·36 videos
Preference Data: The Most Opaque Part of Post-Training | RLHF & Post-training Course, Lecture 8
Posted
July 20, 2026
Views
821
Likes
44
Comments
5
Engagement
July 20, 2026
821
44
5
5.97%
Every word spoken in this episode is indexed. Type any phrase to jump straight to the moment it was said.
Type any word or phrase that may have been spoken. Click a result to seek the player to that exact moment.
Try a name, a topic, or a quoted line
Episode Timeline
Every episode in order around the one you’re watching — click any card and the page flips straight to it.
35:53Now PlayingThis time we dig into the philosophical and practical underbelly of post-training. Discussing how economics, philosophy, and control theory ideas converged into modern RLHF techniques. We also discuss opaque details of modern human data collection. This one is a fun one for me, because I get to link to a lot of papers I really enjoyed writing a few years ago.
Welcome to The RLHF Book & Post-Training Course with Nathan Lambert.
Ask questions and I'll answer them in the next roundup video!
Slides for this lecture are here
Chapters:
Some papers and resources referenced:
The History and Risks of RLHF (Entangled Preferences) Lambert et al., 2023
Objective Mismatch in Modelbased RL Lambert et al., 2020
The Alignment CeilingObjective Mismatch in RLHF Lambert & Calandra, 2023
InstructGPT labeler instructions (PDF)
All resources will be available at
Order a copy of the book (physical recommended) on Manning.com
Order a copy on Amazon
With specific course resources at (recording links, slides in PDF and native form, etc.)
And code at
Get more information on Nathan at and stay up to date with his work on Interconnects
Course YouTube playlist
Join the book's Discord Community
Nathan is on…
Slides are built with Colloquium
Thank you to my many collaborators who helped me learn this information I get to share with the world!
Sentinel Indexing in Progress
Metadata and chapters are available. Claim extraction for this episode is pending.
All video content is delivered via YouTube embedded players in accordance with the YouTube Terms of Service. Sentinel provides research tools that promote discovery and accountability across political media.
