Building Language Models from Scratch
Build and train your own small language model
By the end, you'll train a small character-level transformer on Shakespeare, locally, with the downloadable Python project, and inspect the text it generates. You start smaller: predicting words from simple counts. From there the course adds context and attention, makes one prediction, applies one update, then trains.
Start lesson 1 →The course, lesson by lesson
- Lesson 1
A language model made of counts
Open lesson → - Lesson 2
How transformers use context to predict new text
Open lesson → - Lesson 3
Make your first transformer prediction
Open lesson → - Lesson 4
Teach the model with one update
Open lesson → - Lesson 5
Train your transformer on Shakespeare
Open lesson → - Lesson 6
Generate text and inspect your trained model
Open lesson →
Download the Python project → · More practice with counts · Attention reference
Find your way in
Building Language Models from Scratch · all lessonsFine-tuning a Model for SQL · all lessonsFundamentals of Machine Learning · all lessons