nanoGPT by Andrej Karpathy: Train a GPT Model in 300 Lines of PyTorch (2026 Guide)

Modern AI frameworks bury the neural network behind thousands of lines of object-oriented abstraction. You install Hugging Face Transformers, call a function, and get a model, but you have no idea what actually happens inside. For anyone serious about understanding how large language models work, this black-box approach is a dead end.

nanoGPT, developed by former Tesla AI Director Andrej Karpathy, is the simplest, fastest open source repository for training and fine-tuning medium-sized GPT models. The entire engine lives in just two highly readable files: a 300-line model definition (model.py) and a 300-line training loop (train.py). No corporate bloat, no abstraction layers, no magic. Just pure PyTorch that an intermediate developer can read and understand in a single afternoon.

In this guide, you'll learn what nanoGPT is, who it's for, and how to train your first GPT model from scratch.

What is nanoGPT?

nanoGPT is the simplest, fastest repository for training and fine-tuning medium-sized Generative Pre-trained Transformers (GPTs). Rather than burying the neural network architecture behind thousands of lines of complex, object-oriented abstraction like massive enterprise libraries, nanoGPT exposes the entire engine in just two highly readable files: a 300-line model definition (model.py) and a 300-line training loop (train.py).

Originally designed to be an educational successor to Karpathy's famous minGPT, nanoGPT prioritizes performance alongside simplicity. It leverages PyTorch native optimizations (like Flash Attention) and can comfortably reproduce OpenAI's foundational GPT-2 (124M parameter) model on a single node in a matter of days. It is the ultimate sandbox for understanding exactly how large language models work under the hood.

Who is it for?

Students and AI enthusiasts: Anyone who wants to demystify artificial intelligence and learn how transformers process language from the absolute ground up.
AI researchers: Scientists and engineers needing a minimal, hackable boilerplate to quickly test new architectural ideas, activation functions, or attention mechanisms.
Enterprise data scientists: Teams looking to pre-train custom, highly specialized small language models (SLMs) from scratch on proprietary corporate data without the overhead of massive frameworks.
Hardware tinkerers: Developers wanting to benchmark custom silicon, GPUs, or Apple Mac architectures against a standardized, lightweight training loop.

What makes nanoGPT different from Hugging Face Transformers?

Extreme simplicity: By stripping away the corporate bloat, the codebase can be read and understood by an intermediate Python developer in a single afternoon. Two files, 600 lines total, zero magic.
Hardware agnostic: Out of the box, the code automatically scales from massive 8x NVIDIA A100 clusters (using Distributed Data Parallel) down to Apple Silicon (MPS) or a basic laptop CPU. No configuration changes needed.
Native Flash Attention: Built to utilize PyTorch's native scaled dot-product attention, massively reducing VRAM requirements and accelerating training speeds.
Drop-in GPT-2 compatibility: The framework natively supports loading the original OpenAI GPT-2 weights from Hugging Face, allowing you to instantly begin fine-tuning a pre-trained model instead of starting from scratch.
Free and open source: The entire framework is free and open source. No course tuition, no API fees, no proprietary lock-in. Just your hardware, your electricity, and Karpathy's brilliant code.

What you need before you start

Operating system: Linux (Ubuntu/Debian) or macOS. Windows users should use WSL2 for maximum compatibility.
Hardware: An NVIDIA GPU is highly recommended for actual pre-training. An Apple Silicon Mac (M1/M2/M3) or CPU is sufficient for the tiny Shakespeare datasets.
Software: Python 3.9+ and Git.
Python libraries: torch (PyTorch 2.0+ required for Flash Attention), numpy, tiktoken, datasets, and optionally wandb for logging.

Step-by-step installation

Step 1: Clone the repository

Pull the clean, minimal codebase directly from GitHub:

git clone https://github.com/karpathy/nanoGPT.git
cd nanoGPT

Step 2: Install Python dependencies

It is best practice to install these within a virtual environment or Conda environment to keep your system clean:

pip install torch numpy transformers datasets tiktoken wandb tqdm

Step 3: Prepare the dataset (Tiny Shakespeare)

Before you can train a model, you need to turn text into a stream of integer tokens. We'll start with the fast, character-level Shakespeare dataset:

python data/shakespeare_char/prepare.py

This script downloads the text and generates train.bin and val.bin files containing your tokenized data.

Step 4: Train your baby GPT

Now, execute the training script using the pre-configured settings for the Shakespeare dataset. If you don't have a dedicated GPU, append --device=cpu or --device=mps to the command:

python train.py config/train_shakespeare_char.py

Watch as the loss drops. The model is actively learning the grammatical structure of Shakespeare's writing.

Step 5: Sample the model

Once training completes (or even while you have an intermediate checkpoint), you can ask the model to generate text based on what it learned:

python sample.py --out_dir=out-shakespeare-char

Common errors and how to fix them

Error	What it means	How to fix it
CUDA Out of Memory (OOM)	Your GPU does not have enough VRAM to handle the current batch size or context window (block size).	Open your configuration file and reduce `batch_size` (for example, from 12 to 4) or decrease the `block_size`.
"Flash Attention requires PyTorch >= 2.0"	The code is attempting to use hardware-accelerated attention, but your PyTorch version is outdated.	Upgrade PyTorch via `pip install --upgrade torch`, or if your GPU does not support it, manually set `dropout` > 0.0 to disable Flash Attention fallback.
"Dataset scripts are no longer supported"	The Hugging Face `datasets` library updated its backend, causing older prepare scripts to fail on OpenWebText.	Downgrade the datasets library (`pip install datasets==2.14.0`) or check the repository's open issues for community patch scripts.

nanoGPT vs cloud fine-tuning APIs

Feature	nanoGPT (local and open source)	Cloud fine-tuning APIs (OpenAI API)
Cost structure	Free (only pay for your own hardware and electricity)	High per-token training and hosting fees
Architectural control	Total (change every math operation in the network)	None (strictly black-box training)
Data privacy	100% private (train on your own machine)	Data is uploaded to corporate servers
Ease of use	Requires Python and deep learning knowledge	Upload a JSON file and click a button
Best for	Learning and research	Rapid production deployment

Bottom line: nanoGPT strips away the overwhelming complexity of modern AI frameworks, providing a naked, brutally elegant look at how large language models are built and trained. It is mandatory homework for anyone serious about understanding the future of artificial intelligence. If you want to stop treating GPT as magic and actually understand what happens inside the transformer, this is where you start.

3 alternatives worth checking out

nanochat (github.com/karpathy/nanochat): Released by Karpathy as the spiritual successor to nanoGPT, this updated framework shifts focus from pre-training base text models to training instruction-following conversational agents (like ChatGPT) with modern optimization metrics.
LitGPT (github.com/Lightning-AI/litgpt): A highly optimized, production-ready framework by Lightning AI. It maintains excellent readability but expands its scope to support training and fine-tuning dozens of modern enterprise architectures (like Llama 3, Mistral, and Phi).
Hugging Face Transformers (github.com/huggingface/transformers): The industry standard for deploying AI. While the source code is massively complex and heavily abstracted, it offers unparalleled support for thousands of pre-trained models, tokenizers, and deployment pipelines right out of the box.

Found this guide useful? Check out more AI tools and open source projects on Sudo Scout.