Frontiers in synthetic data

<p>Synthetic data is known to be a super powerful tool for every level of the language modeling stack. It's documented as being used for expanding vanilla pretraining data and creating large swaths of fine-tuning data. Many, many more rumors surround its use, Anthropic's pretraining-scale constitutional AI, Mistral AI's first models being pretrained on OpenAI outputs, Q-star's hopes as OpenAI's remaining moat, and much more. The diversity of use cases for synthetic data makes planning around the role of synthetic data in solving specific goals.<br />This is AI generated audio with Python and 11Labs.<br />Source code: <a class="linkified" href="https://github.com/natolambert/interconnects-tools" target="_blank">https://github.com/natolambert/interconnects-tools</a><br />Original post: <a class="linkified" href="https://www.interconnects.ai/p/frontiers-in-synthetic-data" target="_blank">https://www.interconnects.ai/p/frontiers-in-synthetic-data</a></p><p>00:00 Frontiers in synthetic data<br />01:14 1. Direct distillation is still king<br />02:54 2. Are Gemini Flash and Claude Haiku distilled?<br />04:03 3. Filtering prevents collapse<br />06:30 4. Synthetic data strategy taxes<br />07:32 5. Pros and cons of training on multi-output-source synthetic datasets<br />08:54 6. Structured synthetic data<br />09:42 7. Weak-to-strong generalization is maybe real<br />10:27 8. Creating synthetic prompts is overlooked again</p> <br /><br />This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.interconnects.ai/subscribe?utm_medium=podcast&#38;utm_campaign=CTA_2">www.interconnects.ai/subscribe</a>