Soup CLI: Fine-Tune an 8B LLM on a 4GB Laptop GPU, No Cloud Required (2026 Guide)

Fine-tuning an 8B parameter LLM like Llama-3.1-8B normally requires a GPU with at least 16GB of VRAM. For developers with a 4GB laptop GPU (like an RTX 3050), that means either renting cloud GPUs at $2-4 per hour, or giving up on local fine-tuning entirely. The barrier to entry for custom model training is simply too high for most developers.

Soup CLI, built by Alpamys Makazhan, is a free, open source LoRA fine-tuning tool that lets you train an 8B LLM on a 4GB laptop GPU. It works by keeping the frozen base model in system RAM and streaming it into the GPU one decoder layer at a time. Peak VRAM becomes one layer instead of the whole model. Measured on an RTX 3050 Laptop with 4GB VRAM: Llama-3.1-8B trains at 119.6 tokens per second in 3.32GB peak VRAM. One YAML config, one command. SFT, DPO, GRPO, KTO, plus eval, gating, and export.

In this guide, you'll learn what Soup CLI is, how it works, and how to fine-tune your first model on a budget laptop.

What is Soup CLI?

Soup CLI is an open source LoRA fine-tuning tool that solves the VRAM problem for local LLM training. During LoRA (Low-Rank Adaptation) fine-tuning, the base model is frozen. It is read, never written. So it does not have to live in the GPU. It only has to arrive before the matrix multiplication that uses it. Soup keeps the frozen base model in system RAM and streams it into the GPU one decoder layer at a time.

This means peak VRAM is determined by the size of one decoder layer, not the entire model. For Llama-3.1-8B, that is about 3.32GB. A 4GB laptop GPU can handle it with room to spare.

The hard part was not speed. It was proving it is correct. Streaming fails silently: cut the autograd path and the loss still goes down, because the upper layers keep learning. So every release compares a streamed run against a resident one and requires the logits to match exactly. The developer even borrowed 8 H100s for three days to validate the protocol, which found a bug in the released code where gradients were silently wrong above a certain layer size while the loss curve looked healthy. The bug was published with a reproducer.

Who is it for?

Developers with budget laptops: If you have a 4GB GPU (RTX 3050, GTX 1650, or similar) and want to fine-tune 8B models locally without paying for cloud GPUs, Soup CLI makes it possible.
AI researchers: Researchers who need to iterate on fine-tuning configs locally before spending money on cloud GPU runs. Test your YAML config on a laptop, then scale to a bigger GPU.
Privacy-conscious teams: Fine-tune models on your own hardware. Your training data never leaves your machine. No cloud uploads, no API costs.
Open source model builders: Developers creating specialized domain-specific models (medical, legal, coding) who want to fine-tune locally and export to standard formats.

What makes Soup CLI different from other fine-tuning tools?

4GB VRAM for 8B models: The core innovation. Stream frozen layers from RAM to GPU one at a time. Peak VRAM is one layer (3.32GB for Llama-3.1-8B), not the whole model (16GB+). No other LoRA tool does this.
Correctness protocol: Every release compares streamed-run logits against a resident run and requires exact matches. This catches the silent failure mode where streaming breaks the autograd path but the loss still goes down. No other tool validates this.
Full training pipeline: SFT (supervised fine-tuning), DPO (direct preference optimization), GRPO (group relative policy optimization), KTO (Kahneman-Tversky optimization), plus evaluation, gating, and export. One YAML config, one command.
119.6 tokens per second on a 3050: Measured, not estimated. Every number is published in the repo, including the ones that turned out wrong.
Apache 2.0 license: Free and open source. No subscription, no proprietary lock-in. Every measurement is in the repo, including the failed ones.
Standard export: Export fine-tuned models to standard formats (Hugging Face, GGUF) for deployment with Ollama, llama.cpp, or vLLM.

What you need before you start

GPU: Any NVIDIA GPU with at least 4GB VRAM (RTX 3050, GTX 1650, or similar). The frozen base model lives in system RAM, so you also need enough RAM to hold it (about 16GB for an 8B model).
System RAM: At least 16GB. The base model sits in RAM and streams to the GPU. 32GB recommended for larger models.
Python: Python 3.10+ with pip.
CUDA: CUDA 12.0+ for NVIDIA GPU support.
Storage: Enough disk space for the base model (about 16GB for Llama-3.1-8B) plus your training data.

Step-by-step installation

Step 1: Install Soup CLI

Install Soup CLI via pip:

pip install soup-cli

Step 2: Create a training config

Create a YAML config file that defines your training run:

# config.yaml
model: meta-llama/Llama-3.1-8B-Instruct
method: sft  # or dpo, grpo, kto
dataset:
  path: ./my-training-data.jsonl
  format: chat  # or instruction, raw
training:
  epochs: 3
  batch_size: 1
  learning_rate: 5e-5
  lora_rank: 16
  max_seq_length: 2048
export:
  format: huggingface  # or gguf
  output_dir: ./output

Step 3: Run the training

Start training with one command:

soup train config.yaml

Soup will load the base model into system RAM, stream layers to the GPU one at a time, and train your LoRA adapter. On a 4GB RTX 3050, expect about 119.6 tokens per second for Llama-3.1-8B.

Step 4: Evaluate and export

After training, evaluate the model and export it:

# Evaluate on your test set
soup eval config.yaml --checkpoint ./output/checkpoint-best

# Export to Hugging Face format for Ollama or vLLM
soup export config.yaml --format huggingface --output-dir ./exported

# Or export to GGUF for llama.cpp
soup export config.yaml --format gguf --output-dir ./exported

Common errors and how to fix them

Error	What it means	How to fix it
CUDA Out of Memory (OOM)	Your GPU does not have enough VRAM even for one layer. This should not happen with 4GB+ GPUs for 8B models, but can happen with larger models.	Reduce `max_seq_length` in your config (for example, from 2048 to 1024), or use a smaller model (7B instead of 8B).
System RAM exhausted	The frozen base model does not fit in system RAM. An 8B model needs about 16GB of RAM.	Close memory-heavy applications, or use a smaller model. 32GB+ RAM is recommended for comfortable operation.
Logits mismatch warning	The correctness protocol detected that the streamed run does not match the resident run. This means the streaming is breaking the autograd path.	Report this as a bug on GitHub with your config and model. The developer actively maintains the correctness protocol and publishes fixes.
Training data format error	Your JSONL file does not match the expected format for the chosen training method.	Check the Soup CLI docs for the correct format for SFT vs DPO vs GRPO. Each method expects different fields in the JSONL.

Soup CLI vs cloud GPU fine-tuning

Feature	Soup CLI (local, open source)	Cloud GPU fine-tuning (RunPod, Lambda)
Cost	Free (only your electricity)	$2-4 per hour for an A100, $0.50-1 for consumer GPUs
VRAM needed	4GB (one layer at a time)	16-80GB (entire model in VRAM)
Data privacy	100% local, training data never leaves your machine	Training data uploaded to cloud servers
Speed	119.6 tok/s on a 3050 (slower but usable)	10-50x faster on A100/H100
Iteration speed	Instant, no queue time	Need to spin up instances, wait for availability
Best for	Local iteration, privacy, budget developers	Large-scale training, production models, speed-critical work

Bottom line: Soup CLI is the only tool that lets you fine-tune an 8B LLM on a 4GB laptop GPU. If you have been priced out of LLM fine-tuning because you do not have a $2,000 GPU, this is your solution. The correctness protocol (requiring streamed logits to exactly match resident logits) is what sets it apart from hacky VRAM-reduction tricks that silently produce wrong gradients. 119.6 tokens per second on a 3050 is genuinely useful for local iteration before spending money on cloud GPUs. For developers who want to build custom models without cloud costs, Soup CLI is the best free open source option available.

3 alternatives worth checking out

Unsloth (github.com/unslothai/unsloth): The most popular open source LoRA fine-tuning library. Unsloth is 2x faster than standard Hugging Face training and uses less memory, but it still requires the entire model to fit in VRAM (minimum 8GB for 8B models with 4-bit quantization). If you have 8GB+ VRAM, Unsloth is faster. If you only have 4GB, Soup CLI is your only option.
Axolotl (github.com/OpenAccess-AI-Collective/axolotl): A widely-used fine-tuning tool that supports many methods (SFT, DPO, PPO, ORPO) and model architectures. Axolotl is more feature-rich than Soup CLI but requires more VRAM. Good for developers who have 16GB+ GPUs and want a polished config-driven experience.
PEFT (Hugging Face) (github.com/huggingface/peft): The foundational library that most other LoRA tools (including Soup CLI) build on. PEFT gives you maximum flexibility but requires you to write Python code and manage VRAM yourself. Use PEFT if you want full control and are comfortable writing training loops.

Found this guide useful? Check out more AI tools and open source projects on Sudo Scout.