Q&A 2: Mastering the Derivations, Running Algorithms at Home & Notation Gotcha's | RLHF Course

Catching up on user questions and making sure we dot all the i's and cross the t's in the course. This is all about going deeper on the derivations and notation of doing post-training right. Plus, a bunch of links to more resources (again).

Welcome to The RLHF Book & Post-Training Course with Nathan Lambert.

Ask questions and I'll answer them in the next roundup video!

Slides for this lecture are here

Chapters:

Estimating KL in practice John Schulman's k1/k2/k3 note (essential KLpenalty lore)

Suggested Experiments (each chapter: runnable code + knobs to vary):

• Ch 4 · Instruction tuning

• Ch 5 · Reward models

• Ch 6 · Policy gradients

• Ch 8 · Direct alignment (DPO/IPO easiest start)

• Ch 9 · Rejection sampling

Interactive modelcomparison library (see each posttraining stage on the same prompt)

All resources will be available at

Order a copy of the book (physical recommended) on Manning.com

Order a copy on Amazon

With specific course resources at (recording links, slides in PDF and native form, etc.)

And code at

Get more information on Nathan at and stay up to date with his work on Interconnects

Course YouTube playlist

Join the book's Discord Community

Nathan is on…

Slides are built with Colloquium

Thank you to my many collaborators who helped me learn this information I get to share with the world!