Build an LLM from Scratch 5: Pretraining on Unlabeled Data
March 23, 2025
32,375
654
36
2.13%
Search the Record
IndexedEvery word spoken in this episode is indexed. Type any phrase to jump straight to the moment it was said.
Type any word or phrase that may have been spoken. Click a result to seek the player to that exact moment.
Try a name, a topic, or a quoted line
Sebastian Raschka Episodes Around March 23, 2025
See what was published immediately before and after this episode.
2:36:44Now PlayingBuild an LLM from Scratch 5: Pretraining on Unlabeled Data
Chapters
YouTube Description
as posted by the channelLinks to the book:
Link to the GitHub repository
This video explains how to pretrain a LLM from scratch.
You can find additional bonus materials on GitHub:
Pretraining GPT on the Project Gutenberg Dataset,
PyTorch Performance Tips for Faster LLM Training,
Converting the GPT2 architecture into Llama 2 and Llama 3,
Guests & Subjects Covered
Sentinel Indexing in Progress
Metadata and chapters are available. Claim extraction for this episode is pending.
All video content is delivered via YouTube embedded players in accordance with the YouTube Terms of Service. Sentinel provides research tools that promote discovery and accountability across political media.