Building Language Models from Scratch

Build and train your own small language model

By the end, you'll train a small character-level transformer on Shakespeare, locally, with the downloadable Python project, and inspect the text it generates. You start smaller: predicting words from simple counts. From there the course adds context and attention, makes one prediction, applies one update, then trains.

Start lesson 1 →

The course, lesson by lesson

  1. Lesson 1

    A language model made of counts

    Open lesson →
  2. Lesson 2

    How transformers use context to predict new text

    Open lesson →
  3. Lesson 3

    Make your first transformer prediction

    Open lesson →
  4. Lesson 4

    Teach the model with one update

    Open lesson →
  5. Lesson 5

    Train your transformer on Shakespeare

    Open lesson →
  6. Lesson 6

    Generate text and inspect your trained model

    Open lesson →

Download the Python project → · More practice with counts · Attention reference

Find your way in

Six ways to learn AI · Karpathy study companion

ML reference slides · Python examples