What it does
An LLM routing layer that sits between an application and multiple LLM providers. Instead of hardcoding which model handles which query, it extracts signals from the query, matches it to a task type, and picks the cheapest capable model — complex math goes to reasoning models, simple QA to small local models, vision queries to multimodal models.
- 9 signal extractors: keyword, domain, complexity, language, safety, PII, vision, tool, modality
- Budget strategies:
cheapest_capable,quality_first,balanced,performance_weighted - Adaptive, EMA-based health tracking that auto-disables and recovers degraded models
- Dynamic model registry, hot-reloadable YAML config, OpenAI-compatible
/v1/chat/completions - Real-time dashboard (traces, pipeline, models, stats, sessions, config, datasets) and 11 built-in tools
How it works
Query → Guardrails → Signal extractors → Decision matcher → Capability filter → Budget selection → Model
- Layer 1 — what task? Decision exemplars in
config/config.yamlare embedded withall-MiniLM-L6-v2; the query is matched to the nearest decision centroid by cosine similarity (vLLM-SR style). - Layer 2 — which model? The model pool is filtered to those with every required capability (e.g.
[vision],[reasoning, code]), then the decision's budget strategy picks one. - Layer 3 — how to call? The decision's inference config (temperature, max_tokens, reasoning, thinking tokens) is passed through to the model call.
- Safety guardrails (jailbreak, PII) run first; adaptive routing tracks model health over time.
Key results
| Benchmark | Metric | Value |
|---|---|---|
| RouterArena (overlap, 809 queries) | Routing accuracy | 86.8% |
| RouterArena | Avg routing latency / cost per 1K queries | 15ms / $0.25 |
| VL-RouterBench (14 datasets, 17 VLMs) | Routing accuracy / rank score | 83.8% / 0.80 |
| vLLM-SR baseline, same RouterArena data | Accuracy / latency / cost per 1K | 69.1% / 22ms / $0.31 |
As reported in the repository README.
Tech stack
PythonFastAPI-style serversentence-transformers (MiniLM)YAML configuvHugging Face embeddings