Q&A 2: Mastering the Derivations, Running Algorithms at Home & Notation Gotcha's | RLHF Course
July 1, 2026
644
22
1
3.57%
Search the Record
IndexedEvery word spoken in this episode is indexed. Type any phrase to jump straight to the moment it was said.
Type any word or phrase that may have been spoken. Click a result to seek the player to that exact moment.
Try a name, a topic, or a quoted line
Nathan Lambert Episodes Around July 1, 2026
See what was published immediately before and after this episode.
14:55Now PlayingQ&A 2: Mastering the Derivations, Running Algorithms at Home & Notation Gotcha's | RLHF Course
Chapters
Segments
YouTube Description
as posted by the channelCatching up on user questions and making sure we dot all the i's and cross the t's in the course. This is all about going deeper on the derivations and notation of doing post-training right. Plus, a bunch of links to more resources (again).
Welcome to The RLHF Book & Post-Training Course with Nathan Lambert.
Ask questions and I'll answer them in the next roundup video!
Slides for this lecture are here
Chapters:
Estimating KL in practice John Schulman's k1/k2/k3 note (essential KLpenalty lore)
Suggested Experiments (each chapter: runnable code + knobs to vary):
• Ch 4 · Instruction tuning
• Ch 5 · Reward models
• Ch 6 · Policy gradients
• Ch 8 · Direct alignment (DPO/IPO easiest start)
• Ch 9 · Rejection sampling
Interactive modelcomparison library (see each posttraining stage on the same prompt)
All resources will be available at
Order a copy of the book (physical recommended) on Manning.com
Order a copy on Amazon
With specific course resources at (recording links, slides in PDF and native form, etc.)
And code at
Get more information on Nathan at and stay up to date with his work on Interconnects
Course YouTube playlist
Join the book's Discord Community
Nathan is on…
Slides are built with Colloquium
Thank you to my many collaborators who helped me learn this information I get to share with the world!
Links & Promotions
Guests & Subjects Covered
Sentinel Indexing in Progress
Metadata and chapters are available. Claim extraction for this episode is pending.
All video content is delivered via YouTube embedded players in accordance with the YouTube Terms of Service. Sentinel provides research tools that promote discovery and accountability across political media.