MiniMind: Train Your Own LLM From Scratch in 2 Hours on a Single GPU (2026 Guide)

Most developers use large language models through APIs or high-level libraries like Hugging Face Transformers. The internals are a black box. You call a function, get a response, and never understand what actually happened inside the neural network. If you want to learn how LLMs really work, the options are either expensive courses or reading dense academic papers with no hands-on code.

MiniMind, developed by Jingyao Gong, is a free, open source educational framework that demystifies the entire LLM training pipeline. It enables you to train a GPT-style language model from absolute scratch, including tokenizer training, pre-training, supervised fine-tuning, and RLHF alignment, in about 2 hours on a single consumer GPU (like an RTX 3090) for around $0.50. The smallest model is just 64M parameters, written in pure PyTorch with zero abstraction layers, so you can read and understand every single line.

In this guide, you'll learn what MiniMind is, who it's for, and how to train your first LLM from scratch.

What is MiniMind?

MiniMind is an ultra-lightweight, open source educational framework designed to demystify the entire large language model (LLM) training pipeline. It enables users to train a GPT-style language model from absolute scratch, including tokenizer training, pre-training, supervised fine-tuning, and RLHF alignment, in about 2 hours on a single consumer GPU (like an RTX 3090) for around $0.50.

The smallest model, at just 64M parameters, is drastically smaller than commercial giants like GPT-3, allowing it to run smoothly on virtually any machine. MiniMind is written in pure PyTorch with zero black box abstraction layers, meaning developers can read, understand, and modify every single line of the neural network architecture.

Who is it for?

AI students and beginners: Developers wanting to learn how LLMs actually work under the hood without relying on abstracted libraries like Hugging Face Transformers.
Researchers and educators: Instructors needing a minimal, hackable boilerplate to teach neural network architectures or test new alignment strategies (PPO, GRPO, DPO).
Hardware tinkerers: Enthusiasts wanting to train and deploy tiny local models (SLMs) on edge devices, Raspberry Pis, or older hardware.
Custom model builders: Developers looking to build highly specialized, domain-specific small language models on private offline datasets.

What makes MiniMind different from using Hugging Face Transformers?

Ultra-cheap and fast: Train a 64M parameter chatbot from zero to conversational in just 2 hours on a single GPU. Total cost: about $0.50 in electricity.
Pure white-box PyTorch: No hidden dependencies or highly abstracted API calls. You build the tokenizer, attention mechanisms, and feed-forward networks from scratch. Every line is readable and modifiable.
Modern architecture: The codebase mirrors cutting-edge architectures like Qwen3 and DeepSeek-V3, including Mixture of Experts (MoE) variants, making the lessons learned directly applicable to full-scale models.
End-to-end pipeline: Includes native scripts for pretraining, SFT (supervised fine-tuning), LoRA, DPO, PPO/GRPO (reinforcement learning), and even model distillation. You see the entire lifecycle, not just one stage.
Broad compatibility: Once trained, models can be easily exported to run on popular inference engines like vLLM, llama.cpp, or Ollama, and includes an OpenAI-compatible API server.
Free and open source: The entire framework is free and open source. No course tuition, no API fees, no proprietary lock-in. Just your hardware, your electricity, and your curiosity.

What you need before you start

Operating system: Linux (Ubuntu 20.04+ recommended) or Windows (via WSL2).
Hardware: Minimum 16 GB RAM. An NVIDIA GPU with at least 8 GB to 24 GB VRAM (for example, RTX 3090 or 4090) is highly recommended for training, though inference can run on CPU.
Software: Python 3.10+, CUDA 12.2+, Git, and standard build tools.
Data: The project provides open source datasets via Hugging Face or direct download scripts.

Step-by-step installation

Step 1: Clone the repository

Pull the main repository to your local machine to get the raw Python source code:

git clone https://github.com/jingyaogong/minimind.git
cd minimind

Step 2: Set up the Python environment

Use a virtual environment or Conda to manage dependencies cleanly and avoid conflicts with other AI projects:

conda create -n minimind python=3.10 -y
conda activate minimind
pip install -r requirements.txt

Step 3: Download the datasets

MiniMind provides pre-cleaned datasets for all stages (pre-training, SFT, DPO). Run the provided script or manually download them into the ./data folder:

python download_data.py

If the script is unavailable, download the required .jsonl files directly from the linked Hugging Face dataset repository into your data folder.

Step 4: Pre-train the base model

Start the foundational pre-training phase. This teaches the model basic language structure from the raw text data:

python train_pretrain.py \
  --data_path ./data/pretrain_data.jsonl \
  --model_config ./config/minimind-3.yaml \
  --epochs 2 \
  --batch_size 32 \
  --learning_rate 5e-4

Step 5: Supervised fine-tuning (SFT) and inference

Once pre-training is complete, fine-tune the model to follow instructions and act as a helpful chatbot. After training, you can launch the interactive chat interface:

python train_sft.py \
  --pretrained_model ./checkpoints/pretrain/best.pt \
  --data_path ./data/sft_data.jsonl \
  --epochs 3

python inference.py --model_path ./checkpoints/sft/best.pt

Common errors and how to fix them

Error	What it means	How to fix it
CUDA Out of Memory (OOM)	Your GPU lacks the VRAM to handle the requested batch size or context window during training.	Lower the `batch_size` parameter in your training script, or enable gradient accumulation in the configuration.
"RuntimeError: Expected all tensors to be on the same device"	PyTorch failed to properly move your model weights or input data from the CPU to the GPU.	Ensure you have a compatible CUDA toolkit installed (`print(torch.cuda.is_available())` should return True) and check that your environment isn't forcing a CPU fallback.
*"FileNotFoundError: ./data/.jsonl"**	The training script cannot find the required datasets in the specified directory.	Ensure you have successfully run the download script or manually placed the `.jsonl` files inside the `minimind/data/` folder before launching training.

MiniMind vs commercial AI courses and APIs

Feature	MiniMind (open source)	Commercial AI courses / APIs
Cost	100% free (only hardware and electricity costs, about $0.50)	High monthly API fees or expensive course tuitions
Code transparency	Pure white-box (understand every layer)	Black-box APIs or heavily abstracted libraries
Data privacy	Total privacy (train offline on local hardware)	Data uploads required for cloud training
Primary goal	Educational mastery of modern LLM architecture	Rapid deployment without understanding internals
Best for	Developers who want to deeply understand LLMs	Developers who just want to use LLMs in production

Bottom line: MiniMind is a masterclass in AI architecture. By stripping away the bloat of enterprise frameworks, it allows anyone with a standard gaming GPU to build, train, and deeply understand a complete large language model from scratch in a single afternoon. If you want to stop treating LLMs as magic black boxes and actually understand what happens inside the neural network, this is the best free starting point available.

3 alternatives worth checking out

nanoGPT (github.com/karpathy/nanoGPT): Andrej Karpathy's legendary, minimal PyTorch implementation of GPT-2. It is the spiritual predecessor to MiniMind and focuses heavily on extreme code simplicity for educational pre-training, though it lacks modern RLHF pipelines out of the box.
LitGPT (github.com/Lightning-AI/litgpt): A robust, hackable framework from Lightning AI that scales from educational use to production-grade deployment, supporting the training and fine-tuning of dozens of modern enterprise architectures (like Llama 3 and Mistral).
llama.cpp (github.com/ggerganov/llama.cpp): If your goal is strictly to run (infer) models locally rather than train them from absolute scratch, llama.cpp is the gold standard for high-performance, heavily quantized CPU/GPU inference.

Found this guide useful? Check out more AI tools and open source projects on Sudo Scout.