RLHF: A thin line between useful and lobotomized

<p>Many, many signs of life for preference fine-tuning beyond spoofing chat evaluation tools.<br />This is AI generated audio with Python and 11Labs.<br />Source code: <a class="linkified" href="https://github.com/natolambert/interconnects-tools" target="_blank">https://github.com/natolambert/interconnects-tools</a><br />Original post: <a class="linkified" href="https://www.interconnects.ai/p/how-rlhf-works-2" target="_blank">https://www.interconnects.ai/p/how-rlhf-works-2</a></p><p>00:00 How RLHF works, part 2: A thin line between useful and lobotomized<br />04:27 The chattiness paradox<br />08:09 The mechanism for making models chattier<br />10:42 Next steps for RLHF research</p><p>Fig 1: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/rlhf/img_012.webp" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/rlhf/img_012.webp</a><br />Fig 2: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/rlhf/img_018.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/rlhf/img_018.png</a><br />Fig 3: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/rlhf/img_025.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/rlhf/img_025.png</a></p> <br /><br />This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.interconnects.ai/subscribe?utm_medium=podcast&#38;utm_campaign=CTA_2">www.interconnects.ai/subscribe</a>