AirLLM: Run a 70B LLM on a 4GB GPU, No Quantization, No Cloud (2026 Guide)

Running large language models locally has always been gated by VRAM. A 70B parameter model like Llama 3 needs roughly 140 GB of VRAM to run in full precision, which means you need multiple expensive GPUs (like 2x RTX 4090 or an A100). For most developers, that is simply not affordable. The usual workaround is aggressive quantization (4-bit, 8-bit), which degrades output quality, or using a cloud API, which sends your data to someone else's server.

AirLLM, developed by Gavin Li, is a free, open source inference framework that lets you run 70B parameter LLMs on a single 4GB GPU without quantization, distillation, or pruning. It works by streaming model layers from disk one at a time, so the GPU only ever holds one layer in memory. The same approach lets you run 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on 12GB, and Kimi K3 (2.8 trillion parameters) on under 4GB.

In this guide, you'll learn what AirLLM is, how it works, and how to run your first 70B model on a budget GPU.

What is AirLLM?

AirLLM is an open source inference framework that dramatically reduces the memory requirements for running large language models. Instead of loading the entire model into GPU VRAM at once (which is what standard inference engines like vLLM or Hugging Face Transformers do), AirLLM splits the model into layers and streams them from disk to the GPU one layer at a time.

The key insight is that during inference, only one layer is active at any given moment. Standard inference engines keep all layers in VRAM for speed, but that means you need enough VRAM to hold the entire model. AirLLM trades some speed for a massive reduction in memory: the GPU only needs enough VRAM for a single layer, not the whole model.

This means the VRAM you need depends on the model's layer size, not its total size. A 70B model with 80 layers needs about 4 GB per layer, so it runs on a 4 GB GPU. A 405B model needs about 8 GB per layer. A 671B model needs about 12 GB. And sparse Mixture of Experts (MoE) models like Kimi K3 (2.8 trillion parameters) only load one expert at a time, so they run on under 4 GB.

Who is it for?

• Developers with budget GPUs: If you have an old RTX 3060 with 4GB VRAM or a GTX 1080 with 8GB and want to run 70B models, AirLLM makes it possible. No need to buy a $2,000 GPU.
• Privacy-conscious users: Run massive models entirely on your own hardware. No API calls, no cloud, no data leaving your machine.
• AI researchers: Experiment with massive models (405B, 671B, 2.8T) on standard workstation hardware without needing access to a multi-GPU cluster.
• Homelab and self-hosting enthusiasts: Add a 70B chatbot to your self-hosted stack running on hardware you already own.

What makes AirLLM different from quantization or cloud APIs?

• No quantization, no quality loss: AirLLM runs models in full precision. You get the same output quality as running the model on a multi-GPU cluster. Quantization (4-bit, 8-bit) degrades output quality, especially for complex reasoning tasks. AirLLM avoids this entirely.
• Runs on 4GB VRAM: A 70B model runs on a single 4GB GPU. A 405B model runs on 8GB. DeepSeek-V3 (671B) runs on 12GB. Kimi K3 (2.8 trillion parameters) runs on under 4GB. No other inference engine can do this without quantization.
• One line of code: The same AutoModel.from_pretrained(...) call works for every model. Pass a Hugging Face repo ID and AirLLM handles the rest: layer splitting, disk streaming, and inference.
• Optional 3x speedup with compression: If you want faster inference and are willing to accept a tiny accuracy loss, AirLLM supports block-wise quantization (4-bit or 8-bit) that speeds up inference by 3x. This is optional and off by default.
• Works on macOS: AirLLM supports Apple Silicon (M1/M2/M3/M4) via the MLX framework. Run 70B models on your MacBook.
• Supports every major model: Llama (2/3/3.1/3.3/4), Qwen (1/2/2.5/3 including MoE and FP8), DeepSeek (V2/V3/R1), Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, Yi, and Kimi K3. New models are supported the day they are released.
• Free and open source: The entire framework is free and open source under the Apache 2.0 license. No API fees, no subscription, no cloud dependency.

What you need before you start

• GPU: Any NVIDIA GPU with at least 4GB VRAM (for 70B models) or 8GB (for 405B models). CPU inference is also supported but much slower.
• Disk space: You need enough disk space to store the model weights. A 70B model is about 140GB on disk. A 405B model is about 800GB. An SSD is strongly recommended for acceptable inference speed.
• Python: Python 3.8+ with pip.
• CUDA: CUDA 12.2+ if using an NVIDIA GPU. For macOS, install MLX.
• RAM: At least 16GB system RAM. 32GB+ recommended for larger models.

Step-by-step installation

Step 1: Install AirLLM

Install the AirLLM pip package:

pip install airllm

Step 2: Run a 70B model on a 4GB GPU

Create a Python script and run a 70B model with a single function call:

from airllm import AutoModel

MAX_LENGTH = 128

# One line works for any model. This runs a 70B model on 4GB VRAM.
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# Want to go bigger? Same one line:
# model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")  # 235B, ~3GB VRAM
# model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, ~12GB VRAM

input_text = ["What is the capital of the United States?"]

input_tokens = model.tokenizer(
    input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False
)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True
)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

Step 3: Enable 3x speedup with compression (optional)

If you want faster inference and are willing to accept a negligible accuracy loss, enable block-wise quantization:

model = AutoModel.from_pretrained(
    "garage-bAInd/Platypus2-70B-instruct",
    compression='4bit'  # or '8bit' for less compression
)

This speeds up inference by approximately 3x by reducing the disk loading size. Unlike full quantization, AirLLM only quantizes the weights (not activations), which preserves accuracy better.

Step 4: Run on macOS (Apple Silicon)

AirLLM works on Apple Silicon Macs via the MLX framework:

# Install MLX first
pip install mlx

# Then use AirLLM the same way as on Linux

Common errors and how to fix them

Error	What it means	How to fix it
MetadataIncompleteBuffer / disk space error	You ran out of disk space during the model splitting process. AirLLM splits the model into layers on disk, which temporarily requires extra space.	Clear your Hugging Face cache (~/.cache/huggingface), free up disk space, and retry. You need roughly 2x the model size in free disk space during the first run.
"ValueError: max() arg is an empty sequence"	You are trying to load a non-Llama model (like Qwen or ChatGLM) using the old AirLLMLlama2 class instead of AutoModel.	Use from airllm import AutoModel instead of from airllm import AirLLMLlama2. AutoModel automatically detects the model type.
"401 Client Error... Repo model is gated"	You are trying to download a gated model (like Meta's Llama) without a Hugging Face token.	Create a Hugging Face account, accept the model license, and pass your token: AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf", hf_token='your_token').
"Asking to pad but the tokenizer does not have a padding token"	Some tokenizers do not have a padding token configured.	Turn off padding in the tokenizer call: padding=False. See the example code above.

AirLLM vs quantization vs cloud APIs

Feature	AirLLM (open source)	Quantization (4-bit/8-bit)	Cloud APIs (OpenAI, Anthropic)
Output quality	Full precision, no quality loss	Degraded, especially for reasoning	Full precision (provider's models)
VRAM needed for 70B	4GB	20-40GB (still needs most of the model in VRAM)	0GB (runs on their servers)
Data privacy	100% local, nothing leaves your machine	100% local	Data sent to cloud servers
Cost	Free (only your electricity)	Free (only your electricity)	Pay per token, $20-200/month subscriptions
Inference speed	Slower than full VRAM (disk streaming)	Faster than AirLLM, slower than cloud	Fastest (optimized infrastructure)
Best for	Running massive models on budget hardware with no quality loss	Running models faster if you have enough VRAM	Maximum speed and model quality without hardware concerns

Bottom line: AirLLM is the only tool that lets you run a 70B parameter LLM in full precision on a 4GB GPU. If you have an old graphics card and want to run massive models locally without paying for cloud APIs or buying expensive hardware, this is the solution. The tradeoff is speed (disk streaming is slower than keeping everything in VRAM), but for tasks where output quality matters more than latency (like content generation, code review, or research), AirLLM is unbeatable. And with Kimi K3 support, you can now run a 2.8 trillion parameter model on under 4GB of VRAM, which is genuinely remarkable.

3 alternatives worth checking out

• Ollama (ollama.com): The most popular local LLM manager. Ollama runs models in VRAM with optional quantization, so it is much faster than AirLLM but requires more VRAM. If you have 16GB+ of VRAM, Ollama is the better choice for speed. If you only have 4GB, AirLLM is your only option for 70B models.
• llama.cpp (github.com/ggerganov/llama.cpp): The gold standard for CPU/GPU inference with aggressive quantization. llama.cpp can run 70B models on CPU with 4-bit quantization, but the output quality is lower than AirLLM's full-precision streaming. Use llama.cpp if you want speed and are okay with quantization. Use AirLLM if you want full quality on a tiny GPU.
• vLLM (github.com/vllm-project/vllm): The fastest inference engine for production deployments. vLLM keeps the entire model in VRAM and uses PagedAttention for maximum throughput. If you have the VRAM (140GB+ for a 70B model), vLLM is 10-50x faster than AirLLM. But if you do not have that VRAM, AirLLM is your answer.

---

Found this guide useful? Check out more AI tools and open source projects on Sudo Scout.