Needle 2: The 14MB AI Model That Runs on Phones, Wearables, and Smart Home Devices (2026 Guide)

Running AI models on small devices has always meant choosing between two bad options. Either use a cloud API (which sends user data to a server, requires internet, and costs money per request) or run a quantized 7B model locally (which needs 4GB+ of RAM, drains battery, and is too slow for real-time interaction on phones and wearables). For smart home devices, robots, and IoT, neither option works. They do not have the RAM, the battery, or the connectivity.

Needle 2, built by Cactus Compute, is a free, open source 45M-parameter foundation model designed specifically for tool calling, device use, and structured extraction on tiny devices. The entire model is a single 14MB binary that runs a full session in about 28MB of RAM. It works fully offline, calls your tools, extracts structured data, and carries a calibrated confidence score so you know when to trust it and when to escalate. 5,400+ stars on GitHub.

In this guide, you'll learn what Needle 2 is, how it works, and how to deploy it on tiny devices.

What is Needle 2?

Needle 2 is an open 45M-parameter model for tool calling, device use, and structured extraction. The whole model is a single 14MB binary that runs a full session in about 28MB of RAM. It is built on Cactus Compute's Simple Attention Network architecture, compressed to CQ2-bit with Cactus Quants, and baked into its own engine.

The architecture uses a Hadamard MLP in place of the traditional FFN, grouped query attention (GQA), engram key-value memory, and multi-lane hyper-connections. Each block carries its own update rule with learned, input-dependent gates. Both attention and MLP residuals are sandwich-normed and gated. Decoding is constrained by a byte-level grammar compiled from your declared schemas, so the model can only output valid JSON for your tools.

On benchmarks, Needle 2 trades wins with other small models like FunctionGemma 270M, LFM2.5 230M, and Apple FM, at 5x to 70x smaller, and 2 bits against their f16.

Who is it for?

Mobile and wearable developers: Build apps with on-device AI that calls tools (weather, calendar, smart home controls) without sending data to the cloud or draining battery.
Smart home and IoT builders: Run tool-calling AI directly on ESP32, Raspberry Pi Pico, or smart home hubs. No cloud dependency, no latency, works offline.
Robotics engineers: Give robots a lightweight brain for tool calling and structured decision-making that runs on the robot itself, not in a cloud data center.
Privacy-first app developers: Apps that need AI features (intent parsing, tool selection, data extraction) but cannot send user data to cloud APIs. Needle 2 runs entirely on-device.

What makes Needle 2 different from running a quantized Llama or Gemma?

14MB binary, 28MB RAM: The entire model is a single 14MB file. It runs a full session in about 28MB of RAM. Compare that to a quantized 7B model at 4GB+ RAM. Needle 2 runs on devices that cannot run any other LLM.
Tool calling first: Needle 2 is designed for tool calling, not general chat. You declare your tools (Python functions with type hints and docstrings), and the model decides which tool to call and how to fill arguments. The byte-level grammar constrains every token to valid JSON for your schemas.
Confidence-gated: Every response carries a calibrated confidence score from a learned head. Set a threshold, act above it, escalate below it. This is critical for devices where a wrong tool call could have physical consequences (smart home, robotics).
Tool retrieval: Declare a large catalog of tools and a built-in retrieval head renders only the top five tools per turn, with the grammar constrained to that subset. You can have hundreds of tools without bloating the context.
Bounded memory: A 256-token sliding window with tools pinned as KV sinks, so total memory stays near 28MB no matter how long the conversation runs. No memory leak on long sessions.
LoRA fine-tuning: Fine-tune Needle 2 on your specific tools and use cases with LoRA. The fine-tuned model is still a single .cact file that runs on the same engine. Training runs on any JAX-supported accelerator (NVIDIA GPU, Apple Silicon Metal).
Structured extraction: Pass a Pydantic model and get a typed object back. Declare the shape, call extract(), and Needle 2 pulls structured data out of unstructured text.
Fully offline: Inference does no network calls. The engine and weights are a single binary. Air-gapped deployment is supported. No Hugging Face download at runtime, no API calls, no telemetry.
Free and open source: MIT license. 5,400+ stars. The entire model, engine, fine-tuning pipeline, and playground are open source.

What you need before you start

Python 3.10+: For the cactus-needle package, inference, and fine-tuning.
A tiny device (for deployment): Anything with 28MB+ of RAM. Phones, Raspberry Pi, ESP32 (with modifications), smart home hubs, wearables.
An accelerator (for fine-tuning, optional): NVIDIA GPU with CUDA, or Apple Silicon with Metal. Fine-tuning uses JAX.
OpenRouter API key (optional): Only needed if you want to synthesize training data automatically. Inference and fine-tuning on your own data need no API key.

Step-by-step installation

Step 1: Install Needle

pip install cactus-needle

Step 2: Define your tools and run the agent

Decorate Python functions as tools. The signature gives argument types, the docstring is the tool description, and run() completes the loop:

import needle

@needle.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}

@needle.tool
def set_lights(room: str, brightness: int):
    "Set the brightness of lights in a room."
    return {"status": "ok", "room": room, "brightness": brightness}

agent = needle.Needle(tools=[get_weather, set_lights])
result = agent.run("dim the kitchen to 10")
print(result["results"])
# [{'room': 'kitchen', 'brightness': 10, 'status': 'ok'}]

Step 3: Use confidence gating

Every response carries a confidence score. Set a threshold to decide when to act and when to escalate:

result = agent.run("turn off all the lights")
if result["confidence"] > 0.8:
    # Act on the tool call
    execute_tool(result["results"])
else:
    # Escalate to a larger model or ask the user for confirmation
    escalate(result)

Step 4: Extract structured data

Pass a Pydantic model and get a typed object back:

from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

invoice = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice)
print(invoice.vendor, invoice.total)  # -> Acme Corp 1200.0

Step 5: Fine-tune on your tools (optional)

Create a JSONL file with examples, then fine-tune with LoRA:

# Fine-tune on your data
needle finetune data.jsonl --epochs 10 --lora-rank 16 --lora-alpha 32

# Build a tuned .cact file
needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact

# Run the tuned model
python -c "import needle; agent = needle.Needle(weights='my_needle.cact', tools=[...]); print(agent.run('...'))"

Step 6: Try the playground

needle playground  # Opens at http://127.0.0.1:7860

Edit tools and prompts in the browser, run queries, and fine-tune from the UI.

Common errors and how to fix them

Error	What it means	How to fix it
Model download failed	The Needle 2 engine could not be downloaded from Hugging Face.	Check your internet connection. The engine is fetched once and cached. For air-gapped deployment, manually download the engine and place it in the cache directory.
Tool call JSON invalid	The model produced JSON that does not match your tool schema.	Ensure your tool schemas are correctly typed. The byte-level grammar should prevent this, but if it happens, check that your Pydantic models or JSON schemas are valid.
Fine-tuning OOM	Your GPU does not have enough VRAM for the LoRA training.	Reduce `--lora-rank` (default 16, try 8), reduce `--batch-size` (default 16, try 4), or reduce `--max-len` (default 1024, try 512).
Low confidence on all outputs	The model is not confident enough for your use case.	Fine-tune on your specific tools and data. The base model is general-purpose. Fine-tuning on 100-500 examples typically improves confidence significantly.

Needle 2 vs quantized Llama vs cloud APIs

Feature	Needle 2 (on-device)	Quantized Llama 7B (on-device)	Cloud APIs (OpenAI, Anthropic)
Model size	14MB	4GB+ (4-bit quantized)	0 (runs on their servers)
RAM needed	28MB	4GB+	0
Internet required	No (fully offline)	No	Yes (always)
Data privacy	100% on-device	100% on-device	Data sent to cloud
Tool calling	Native, grammar-constrained	Possible but not optimized	Native
Confidence score	Yes (calibrated)	No	No
Cost	Free	Free (your electricity)	Pay per token
Best for	Tiny devices, offline, privacy, tool calling	General chat on capable hardware	Maximum quality, no hardware constraints

Bottom line: Needle 2 is the only model that makes on-device AI tool calling practical for phones, wearables, smart home devices, and robots. At 14MB and 28MB RAM, it runs on hardware where no other LLM can. The confidence gating is the killer feature for physical devices: you set a threshold, the model acts above it, and escalates below it. That is the safety mechanism that makes on-device AI viable for smart home and robotics. If you are building apps or devices that need AI tool calling without cloud dependency, Needle 2 is the best free open source option available.

3 alternatives worth checking out

Ollama (ollama.com: The most popular local LLM manager. Ollama can run small models (1B-3B) on phones and Raspberry Pis, but even the smallest models need 1-2GB of RAM. If your device has 2GB+ RAM, Ollama is easier to use. If your device has less than 512MB, Needle 2 is your only option.
Apple Foundation Models (developer.apple.com/machine-learning): Apple's on-device models built into iOS 18+. These are excellent but locked to Apple devices. If you are building for iOS only, Apple FM is the best choice. If you need cross-platform (Android, IoT, Linux), Needle 2 is the open alternative.
Llama.cpp (github.com/ggerganov/llama.cpp: The gold standard for running quantized models on CPU. Llama.cpp can run very small models (like TinyLlama 1.1B) on phones, but even at 4-bit quantization, 1B models need 600MB+ of RAM. Needle 2 at 14MB is 40x smaller. Use Llama.cpp for general chat, Needle 2 for tool calling on tiny devices.

Found this guide useful? Check out more AI tools and open source projects on Sudo Scout.