The Morning Dev

Only the news worth knowing

‹
 

Watch · Deep learning from first principles

Let's reproduce GPT-2 (124M)

Chapters

  1. 0:00intro: Let’s reproduce GPT-2 (124M)
  2. 3:39exploring the GPT-2 (124M) OpenAI checkpoint
  3. 13:47SECTION 1: implementing the GPT-2 nn.Module
  4. 28:08loading the huggingface/GPT-2 parameters
  5. 31:00implementing the forward pass to get logits
  6. 33:31sampling init, prefix tokens, tokenization
  7. 37:02sampling loop
  8. 41:47sample, auto-detect the device
  9. 45:50let’s train: data batches (B,T) → logits (B,T,C)
  10. 52:53cross entropy loss
  11. 56:42optimization loop: overfit a single batch
  12. 1:02:00data loader lite
  13. 1:06:14parameter sharing wte and lm_head
  14. 1:13:47model initialization: std 0.02, residual init
  15. 1:22:18SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
  16. 1:28:14Tensor Cores, timing the code, TF32 precision, 333ms
  17. 1:39:38float16, gradient scalers, bfloat16, 300ms
  18. 1:48:15torch.compile, Python overhead, kernel fusion, 130ms
  19. 2:00:18flash attention, 96ms
  20. 2:06:54nice/ugly numbers. vocab size 50257 → 50304, 93ms
  21. 2:14:55SECTION 3: hyperpamaters, AdamW, gradient clipping
  22. 2:21:06learning rate
Show the full video description ›

We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Links: - build-nanogpt GitHub repo, with all the changes in this video as individual commits: https://github.com/karpathy/build-nanogpt - nanoGPT repo: https://github.com/karpathy/nanoGPT - llm.c repo: https://github.com/karpathy/llm.c - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - Attention is All You Need paper: https://arxiv.org/abs/1706.03762 - OpenAI GPT-3 paper: https://arxiv.org/abs/2005.14165 - OpenAI GPT-2 paper: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf- The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: https://lambdalabs.com Chapters: 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 parameter sharing wte and lm_head 01:13:47 model initialization: std 0.02, residual init 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms 01:39:38 float16, gradient scalers, bfloat16, 300ms 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms 02:00:18 flash attention, 96ms 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping 02:21:06 learning rate