AI Model Routing
Definition
AI model routing is a serving-layer pattern where a router sits in front of multiple models and decides how to handle each request — not just which model to call, but whether to orchestrate collaboration between models behind a single stable API surface. The router becomes a control plane for inference, turning one API call into a bounded micro-agent runtime.
Core Ideas
From Selection to Collaboration
Traditional routers pick one model per request (cost optimization, safety filtering, latency). The next generation of routers can:
- Cut cost — route simple requests to cheaper models
- Enforce safety — send sensitive domains to stricter models/filters
- Coordinate cloud and edge — keep private intent local, escalate harder work
- Make the model better — turn one call into a bounded collaboration
The key insight: collaboration should be an open serving primitive, not locked inside one commercial endpoint or application-specific agent graph.
Micro-Agent as Serving Runtime
A micro-agent is not an unbounded autonomous agent. It is a small runtime with:
- Budget — cost and token limits
- Topology — which models participate and in what roles
- Trace — full observability of internal collaboration
- Failure policy — what happens when a participant fails
The user sees one model name (e.g., vllm-sr/auto). Behind it, the router selects a recipe based on task signals.
Collaboration Patterns (Looper Algorithms)
Confidence
Sequential escalation. Try a cheaper model first, measure confidence (logprob, self-verification, entailment). Escalate only if the score is too low. Saves cost on easy requests.
Ratings
Bounded parallel fan-out. Launch several candidates up to a max_concurrent cap. Aggregate with rating-aware weights. Useful for A/B evaluation and ensemble strategies.
ReMoM (Repeated Mixture-of-Models)
Fan out breadth samples, wait for minimum-success quorum, run a synthesis round that merges evidence into the required output contract. Falls back to best valid evidence if synthesis fails.
Fusion
Panel → Judge → Finalizer. Independent panel answers become evidence. The judge sees agreement, contradiction, and unique insight. Especially useful for hard reasoning tasks where disagreement is signal.
Workflows
Most agentic pattern with strictest boundaries. Planner assigns roles (e.g., patcher, verifier, finalizer). Bounded by max steps, parallelism, timeouts. Useful for SWE-style tasks without application-owned agent stacks.
Key Principle: Recipes Are Task-Shaped
No single algorithm always wins. The router selects the collaboration pattern based on task characteristics:
- Hard science multiple-choice → ReMoM with strict answer preservation
- Code generation → code-shaped loop detecting constraints, starter code, hidden-test risk
- Formal reasoning with disagreement risk → deeper ReMoM or Fusion
- SWE tasks → Workflows with planner + patcher + verifier
A recipe defines: model pool, model roles, reasoning effort, concurrency, quorum, timeout, synthesis model, fallback policy, output contract, and observability labels.
Notable Implementations
- vLLM Semantic Router — open serving layer implementing all looper patterns behind
vllm-sr/auto - Sakana Fugu — commercial product where a “model” is a surface with a team behind it
- Research: Conductor, Trinity
Benchmark Results (vLLM SR)
| Benchmark | Score | vs Frontier |
|---|---|---|
| LiveCodeBench (Jan-Apr 2025) | 92.6 | Fugu Ultra 92.0, GPT-5.5 90.7 |
| GPQA-Diamond | 96.0 | Fugu Ultra 95.5, Gemini 3.1 Pro 94.3 |
| Humanity’s Last Exam | 50.0 | Fugu Ultra 50.0, Gemini 3.1 Pro 45.0 |
Router-owned collaboration can create a stronger model identity than individual calls beneath it.
Related
- Coding Agents and AI Tools
- Reasoning Effort Control — automatic effort selection is the routing decision applied to a single model instead of across models
Source References
- Micro-Agent: Beat Frontier Models with Collaboration inside Model API