Model merging lessons in The Waifu Research Department

<p>Note: some of the audio in the second half is a little wonky, but the general voice was upgraded so hopefully it's a little less "poppy" until then!<br />I'm trying to fix little pronunciation problems on a weekly basis. Thanks to my early fans! It'll keep improving. E.g. some of the months were wonky.</p><p>When what seems like pure LLM black magic is actually supported by the literature.<br />This is AI generated audio with Python and 11Labs<br />Source code: <a class="linkified" href="https://github.com/natolambert/interconnects-tools" target="_blank">https://github.com/natolambert/interconnects-tools</a><br />Original post: <a class="linkified" href="https://www.interconnects.ai/p/model-merging" target="_blank">https://www.interconnects.ai/p/model-merging</a></p><p>00:00 Model merging lessons in The Waifu Research Department<br />02:21 How and why does model merging work?<br />07:13 Aside: merging vs. ensembles vs. mixture of experts<br />08:21 Why are people doing this?<br />11:22 Tools &amp; Links<br />11:51 Brief (visual) literature review<br />12:07 Full model merging and recent methods<br />15:55 Weight averaging during pretraining<br />17:18 LoRA merging<br />17:53 More background</p><p>Figure 1: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_005.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_005.png</a><br />Figure 2: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_016.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_016.png</a><br />Figure 3: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_042.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_042.png</a><br />Figure 4: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_051.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_051.png</a><br />Figure 5: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_055.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_055.png</a><br />Figure 6: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_058.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_058.png</a><br />Figure 7: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_060.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_060.png</a><br />Figure 8: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_062.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_062.png</a><br />Figure 9: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_065.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_065.png</a><br />Figure 10: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_075.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_075.png</a><br />Figure 11: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_077.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_077.png</a><br />Figure 12: <a class="linkified" href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_084.png" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/model-merging/img_084.png</a></p> <br /><br />This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.interconnects.ai/subscribe?utm_medium=podcast&#38;utm_campaign=CTA_2">www.interconnects.ai/subscribe</a>