Why reward models are still key to understanding alignment

<p>In an era dominated by direct preference optimization and LLMasajudge, why do we still need a model to output only a scalar reward?<br />This is AI generated audio with Python and 11Labs. Music generated by Meta's MusicGen.<br />Source code: <a class="linkified" href="https://github.com/natolambert/interconnects-tools" target="_blank">https://github.com/natolambert/interconnects-tools</a><br />Original post: In an era dominated by direct preference optimization and LLM-as-a-judge, why do we still need a model to output only a scalar reward?</p><p>Podcast figures:<br />Figure 1: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/reward-models/img_004.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/reward-models/img_004.png</a><br />Figure 2: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/reward-models/img_009.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/reward-models/img_009.png</a></p><p>0:00 Why reward models are still key to understanding alignment</p> <br /><br />This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.interconnects.ai/subscribe?utm_medium=podcast&#38;utm_campaign=CTA_2">www.interconnects.ai/subscribe</a>