nanochat
Karpathy's open-source "teaching ChatGPT" — train a chat model from scratch for under $100
Karpathy's open-source "teaching ChatGPT": train a chat-capable model from scratch for under $100 — the best LLM educational code.
Open foundation models — the base layers everything builds on.
Key features
- Minimal hackable harness for training LLMs end-to-end
- Covers tokenization, pretraining, finetuning, evaluation, inference
- Train a GPT-2 capability model for about $48 on an 8xH100 node
- Single --depth dial auto-tunes all other hyperparameters
- Time-to-GPT-2 leaderboard with reference speedrun script
Good for
- Learning and experimenting with LLM training
- Reproducing GPT-2 training cheaply
Stack & tags
Pythonllm-trainingeducationopen-sourcegpt-2speedrunharnessresearch
Quick start
bash runs/speedrun.shFrom the project README on GitHub.
Screenshots
Screenshots from the project’s GitHub README.