Regularization in RL, Why RL Generalizes, and Why SFT Forgets | RLHF & Post-training Book Lecture 10

Hello! We have got some more math to cover. Again we use KL, both to keep our RL runs from over-optimizing reward models (and other targets, but mostly reward models -- reward models are much more prone to overopt). KL also is a useful lens for understanding why RL generalizes better than SFT, i.e. the shape of the loss. Thanks for watching! This is a pretty advanced one, but probably is still useful.

Welcome to The RLHF Book & Post-Training Course with Nathan Lambert.

Ask questions and I'll answer them in the next roundup video!

Slides for this lecture are here (and cleaned up a bit since the lecture)

Chapters:

All resources will be available at

Order a copy of the book (physical recommended) on Manning.com

Order a copy on Amazon

With specific course resources at (recording links, slides in PDF and native form, etc.)

And code at

Get more information on Nathan at and stay up to date with his work on Interconnects

Course YouTube playlist

Join the book's Discord Community

Nathan is on…

Slides are built with Colloquium

Thank you to my many collaborators who helped me learn this information I get to share with the world!