NVIDIA Switchyard: Route LLM Traffic Across Models and Providers Without Changing Your API (2026 Guide)

Using multiple LLM providers means juggling different API formats. Claude Code speaks Anthropic Messages API. Codex speaks OpenAI Chat Completions. Your custom agent might speak OpenAI Responses. If you want to route traffic between providers (for cost optimization, A/B testing, or fallback), you need to translate between these formats yourself. Most teams build a custom proxy, which is fragile, hard to maintain, and does not handle streaming correctly.

NVIDIA Switchyard, developed by the NVIDIA NeMo team, is a free, open source Rust proxy and library for LLM traffic routing. It translates between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats, routes requests across providers with configurable algorithms (random, LLM classifier, stage router, escalation), records Prometheus metrics, and lets you point coding agents like Claude Code or Codex at any OpenAI-compatible endpoint. Apache 2.0 license.

In this guide, you'll learn what Switchyard is, how it works, and how to route your LLM traffic across providers.

What is NVIDIA Switchyard?

Switchyard is a Rust proxy and library for LLM traffic. It routes requests across providers, translates between OpenAI and Anthropic APIs, records operational metrics, and provides typed, composable routing algorithms.

The core idea is simple: point a coding agent such as Claude Code or Codex at an open-source model. Switchyard translates between the OpenAI Chat, Anthropic Messages, and OpenAI Responses formats, so the agent keeps speaking its native API while the request is served by vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint. The same proxy can spread traffic across several models for A/B benchmarking, apply signal-driven stage routing, or run a custom algorithm you write yourself.

Clients keep their native OpenAI or Anthropic API format. Switchyard picks a configured backend, forwards the request in that backend's own format, and translates the response back into the shape the client expects. The server accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Each configured LLM client selects one upstream format.

Who is it for?

Teams using multiple LLM providers: If you use Claude for complex reasoning and a cheaper model (like DeepSeek or Qwen) for simple tasks, Switchyard routes automatically based on the request content.
AI coding agent users: Point Claude Code or Codex at Switchyard instead of directly at a provider. Switchyard can route simple turns to a cheap local model and complex turns to a premium API, cutting costs significantly.
LLM benchmarking teams: Use the random routing strategy to split traffic across models for A/B testing. Switchyard records Prometheus metrics for requests, errors, latency, tokens, and routing overhead.
Rust developers building LLM infrastructure: The switchyard-libsy crate embeds routing algorithms in your own Rust application. It never calls a model itself: an algorithm decides which target to use and hands every model call back to you.

What makes Switchyard different from LiteLLM or custom proxies?

Protocol translation built in: Switchyard translates between OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses formats. Your Claude Code agent speaks Anthropic, but the backend can be an OpenAI-compatible vLLM endpoint. Switchyard handles the translation in both directions, including streaming.
Four routing strategies: Random (for A/B testing), LLM Classifier (request content decides whether a turn needs the weak or strong tier), Stage Router (signals already in the conversation, like tool results and errors, route most turns without an extra model call), and Escalation Router (every turn runs on the weak tier first, a judge reads the answer to decide whether to send the same request to the strong tier).
Three integration paths: Launcher (run Claude Code, Codex CLI, or OpenClaw through Switchyard with one command), Server (standalone Rust proxy), and Library (embed routing algorithms in your own Rust app).
Prometheus metrics: Built-in metrics cover requests, errors, latency, tokens, and routing overhead. Monitor your routing decisions and model performance in real time.
Rust performance: The core proxy is written in Rust. No Python overhead, no GIL, no garbage collection pauses. It is a single binary that handles translation and routing at wire speed.
Apache 2.0 license: Free and open source. No vendor lock-in, no proprietary protocol. Built by the NVIDIA NeMo team with 22 contributors.

What you need before you start

For the launcher path: uv package manager and an OpenRouter API key (or any OpenAI-compatible endpoint).
For the server path: Rust with Cargo to install the standalone binary.
For the library path: A Rust project where you want to embed routing algorithms.
LLM API keys: At least one provider API key (OpenAI, Anthropic, OpenRouter, or a local model via vLLM/Ollama).
A coding agent (optional): Claude Code, Codex CLI, or OpenClaw if you want to use the launcher path.

Step-by-step installation

Path A: Launcher (run coding agents through Switchyard)

Install uv if you do not have it, then install Switchyard:

curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
uv tool install --python 3.10 "nemo-switchyard[cli]"

Set your API key and launch a coding agent through Switchyard:

export OPENROUTER_API_KEY="your-openrouter-key"

# Launch Claude Code through Switchyard
switchyard launch claude --model switchyard

# Launch Codex through Switchyard
switchyard launch codex --model switchyard

# Launch OpenClaw through Switchyard
switchyard launch openclaw --model switchyard

The agent now routes through Switchyard, which translates between API formats and routes to the configured backend.

Path B: Server (standalone proxy)

Install the standalone Rust binary:

cargo install --locked switchyard-server

Create a routes.toml configuration file (see the Getting Started guide for the full format), then validate and start:

export OPENROUTER_API_KEY="your-openrouter-key"

# Validate your config
switchyard-server --config routes.toml --dry-run

# Start the server
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

Verify the proxy is running:

curl http://localhost:4000/health

Point your agent or application at http://localhost:4000 instead of the provider's API directly.

Path C: Library (embed in your Rust app)

Add Switchyard to your Cargo dependencies:

[dependencies]
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }

The library never calls a model itself. An algorithm decides which target to use and hands every model call back to you, so it drops into an existing proxy, gateway, or agent runtime without owning an HTTP stack.

Common errors and how to fix them

Error	What it means	How to fix it
"Authentication headers in extra_headers rejected"	Switchyard rejects authentication headers passed through `extra_headers` for security.	Pass API keys through the configured environment variables or the route configuration, not through extra headers.
"Route not found"	The model ID in your request does not match any configured route.	Check your `routes.toml` configuration. Ensure the model ID your agent sends matches a route ID in the config. Run `--dry-run` to validate.
"Translation error: unsupported format"	The request format is not supported by the target backend.	Ensure your route configuration specifies the correct upstream format for each target. Switchyard supports OpenAI Chat, Anthropic Messages, and OpenAI Responses.
"uv: command not found"	The `uv` package manager is not installed.	Install uv with `curl -LsSf https://astral.sh/uv/install.sh

NVIDIA Switchyard vs LiteLLM vs custom proxy

Feature	Switchyard (NVIDIA)	LiteLLM	Custom proxy
Language	Rust (wire speed)	Python (GIL overhead)	Varies
Protocol translation	OpenAI Chat, Anthropic Messages, OpenAI Responses	OpenAI format only	You build it
Routing strategies	4 (random, classifier, stage, escalation)	Basic (round-robin, fallback)	You build it
Prometheus metrics	Built in	Via callback	You build it
Coding agent launcher	Claude Code, Codex, OpenClaw	No	No
Library embedding	Rust crate	Python library	N/A
License	Apache 2.0	MIT	Varies
Best for	Teams routing between providers with protocol translation	Teams standardizing on OpenAI format	Teams with very specific custom needs

Bottom line: NVIDIA Switchyard solves the multi-provider LLM routing problem properly. Instead of building a custom proxy that breaks on streaming edge cases or fails to translate between API formats, Switchyard handles the translation, routing, and metrics in a single Rust binary. The launcher path is the easiest entry point: one command and your Claude Code or Codex agent is routing through Switchyard. The stage router strategy is particularly clever: it uses signals already in the conversation (tool results, errors) to route most turns without an extra model call, which means lower latency and lower cost. If you are using multiple LLM providers and want to optimize cost, benchmark models, or add fallback, Switchyard is the best free open source proxy available.

3 alternatives worth checking out

LiteLLM (github.com/BerriAI/litellm): The most popular open source LLM proxy. LiteLLM standardizes 100+ LLM providers to the OpenAI format. It is Python-based, widely adopted, and has a managed cloud option. If you only need OpenAI format standardization, LiteLLM is simpler. If you need Anthropic-to-OpenAI translation, routing strategies, and Rust performance, Switchyard is more capable.
OpenRouter (openrouter.ai): A commercial LLM routing service. OpenRouter handles provider routing, fallback, and cost optimization as a managed API. If you want zero infrastructure and do not mind a commercial intermediary, OpenRouter is the easiest option. If you want self-hosted, open source routing, Switchyard is the alternative.
Portkey (portkey.ai): A commercial AI gateway with routing, caching, fallback, and observability. Portkey is feature-rich but proprietary. If you need production-grade routing with a dashboard and do not mind paying, Portkey is excellent. If you want a free, open source, self-hosted alternative, Switchyard covers the core routing and translation needs.

Found this guide useful? Check out more AI tools and open source projects on Sudo Scout.