What Is Routerly? The Beginner's Guide to the Self-Hosted LLM Gateway
CodexFuse Team · 2026-11-05T01:52:30.077Z · 11 min read min read
What is Routerly?
Routerly is a self-hosted LLM gateway that sits between your applications and the large language models they call, deciding where each request goes and how much it may cost. Its own tagline says it best: "One gateway. Any AI model. Total control." You run it on your own hardware — even a small VPS hosting box — and every AI client in your team points at Routerly instead of talking to OpenAI, Anthropic, or Google directly.
That single gateway handles three jobs most teams otherwise bolt together by hand: intelligent routing (each request is scored against pluggable policies and sent to the best available model), real-time cost tracking (token-level pricing per model and per project), and budget enforcement (hard limits that block overspending before it happens). All of it is configured through a web dashboard or CLI, and the whole thing needs no database at all.
Why would you put a gateway between your app and the model?
The short answer is that LLM bills are unpredictable and provider outages are inevitable, and both problems hit hardest when every app in your team has its own API keys and its own base URL. One coding-agent session can chew through hundreds of dollars of tokens while nobody is watching, and one rate limit on a Tuesday afternoon can stall every tool in the company at once.
A Routerly self-hosted gateway attacks both problems from a single control point. Because every request flows through the gateway, Routerly sees the token counts, the cost, and the error rate of every call in one place. When a provider fails or throttles, Routerly's automatic fallback sends the request to the next-best model instead of failing your user's request. And because the gateway speaks the OpenAI and Anthropic API wire formats, your existing tools keep working without modification.
How does Routerly decide which model answers?
Routerly scores every incoming request against a set of pluggable routing policies and sends it to the best available model, falling back automatically when a provider fails. The Routerly LLM gateway currently ships eleven policies — note that the vendor's Docker Hub page still says nine, which is outdated marketing copy — plus four ready-made profiles.
| Policy | What it optimizes for |
|---|---|
llm | Asks a language model to pick the right model for the request |
semantic-intent | Classifies the request's intent with embeddings, then matches a model |
model-preference | Routes to an explicitly preferred model or model list |
cheapest | Picks the lowest-cost model that can handle the request |
health | Avoids providers that are erroring or degraded right now |
performance | Favors low latency and high throughput |
capability | Matches model capability to the difficulty of the task |
context | Considers available context window size |
budget-remaining | Steers traffic based on how much budget is left |
rate-limit | Avoids providers close to their rate limits |
fairness | Distributes requests fairly across the model pool |
The four built-in profiles — auto, cheap, fast, and coding — bundle these policies for common situations, and the vendor's benchmark post uses the llm policy for its accuracy claims. An orchestrator layer can also forward to a priority-ordered list of routers, and a passthrough mode forwards the client's own credential straight to OpenAI or Anthropic for cases where you do not want routing at all.
How does Routerly keep your bill from exploding?
Routerly enforces budgets at three levels — Global, Router, and Per-token — plus an org-and-team tier on top, and an exhausted budget returns an HTTP 503 error before any provider call is made. In Routerly budget enforcement, the cascade reads: Organisation group to Team group to Router to per-token budget, and a child group's limit can never exceed its parent's matching limit.
Limits can be set hourly, daily, weekly, monthly, or yearly, with rolling windows as an option, and they are tracked against cost, call count, input tokens, output tokens, and total tokens. Per-token budgets have three modes — replace, extend, or disable — so you can tune how tightly a single API key is leashed. When a budget is exhausted, the client gets a 503 with a budget_exceeded error, which means no provider call happens and no money is spent.
The dashboard shows spend, call volume, error rates, provider health, and a per-model usage leaderboard in real time, and a notification system — Slack, Teams, PagerDuty, Discord, and email — can alert you as budgets fill up. That combination of hard limits plus alerts is the whole point: the gateway stops the spend first and asks questions later.
How do you set up Routerly?
You can get a gateway running in about five steps. The exact commands below are the vendor's documented routes, checked as of early November 2026.
- Install it. On macOS or Linux, run the vendor's one-liner:
curl -fsSL https://www.routerly.ai/install.sh | bash -s -- --version v0.5.3. Alternatively, run it as a container:docker run -d --name routerly -p 3000:3000 -v routerly_data:/data -e ROUTERLY_HOME=/data --restart unless-stopped inebrio/routerly:latest. A PowerShell one-liner and docker-compose route also exist. - Open
http://localhost:3000/dashboardin your browser and create the admin account. This is the first-run screen the vendor's quick start points you at. - Add your first model with the CLI:
routerly model add, which walks you through provider choice and API key. - Create a router with
routerly router createand pick a policy profile such ascodingorcheapfor your coding-agent traffic. - Point your client at the gateway by changing one environment variable — the base URL. The vendor's line is: "Drop-in compatible with the OpenAI and Anthropic APIs — change one environment variable, nothing else in your code changes." Then set your budget limits in the dashboard before the team starts spending.
Everything the gateway knows — configuration and usage data — lives in plain JSON files under the data directory. In the vendor's words: "config and usage data live in plain JSON files. No PostgreSQL, no Redis, no migrations." That is genuinely unusual for a gateway with this feature list, and it makes backups a simple file copy.
Which clients work with Routerly?
Routerly OpenAI compatible and Anthropic compatible by design — the gateway speaks the standard chat completions and messages wire formats, so anything that speaks the OpenAI or Anthropic protocol works without modification — Cursor, Continue.dev, Open WebUI, LibreChat, LangChain, LlamaIndex, plain curl, and CLI coding tools. That compatibility extends to the newer endpoints: OpenAI-style chat completions and responses formats, and Anthropic's messages format, with streaming support.
If you run Claude Code as your coding agent, the pattern is the one described above: keep the client unchanged and aim its base URL at your Routerly instance, then let a coding profile decide whether each request goes to the strongest model or the cheapest one that can handle it. Two name cautions, each said once: routerly.io is a completely different product (it manages Teltonika and OpenWrt router fleets, not language models), and Routly AI — one letter different — is a separate enterprise LLM-routing product. Neither is related to Inebrio's Routerly.
Routerly vs LiteLLM vs OpenRouter: which one do you need?
These three products get compared constantly, but they are different tools for different jobs — pick by where you want control to live. Our own guides cover LiteLLM and OpenRouter in depth; here is how Routerly positions against them, based on the vendor's own comparison table (last verified by the vendor on 2026-10-21, so treat third-party details as vendor claims).
| Routerly | LiteLLM | OpenRouter | |
|---|---|---|---|
| What it is | Self-hosted gateway product with dashboard, budgets, and 2FA | Open-source Python library for calling many models from code | Hosted API proxy — one key for many models, billed by them |
| Where it runs | Your hardware (Docker) | Your code (import it) | Their cloud |
| Routing intelligence | Eleven composable policies, scored per request | Your own routing logic in code | Automatic model selection tiers |
| Budget enforcement | Hard hierarchical limits, org to API key | You build it | Credit balance, per-key limits |
| Database | None — plain JSON files | None — in-process | Hosted service |
| Best when | A team needs one gateway with spend control and audit | A developer wants multi-model access inside their app | You want zero infrastructure and pay-as-you-go |
The fair summary is that Routerly vs LiteLLM is a product-versus-library question — if you want a running gateway with a dashboard, you want Routerly; if you want multi-model calls inside your own code, you want LiteLLM. Routerly vs OpenRouter is a self-hosted-versus-hosted question: OpenRouter is easier because there is nothing to run, while Routerly keeps your keys, logs, and routing decisions on infrastructure you control. None of these products are rivals in the sense that choosing one forbids the others; teams routinely pair a gateway with a hosted fallback.
What does Routerly cost, and what license is it?
Routerly is AGPL-3.0 licensed — the GitHub repository's license field reads "GNU Affero General Public License v3.0", which confirms what earlier research had flagged as unverified. The maintainer, Carlo Satta, writes: "I maintain Routerly (self-hosted, AGPL, no paid tier), where routing is a set of composable policies and each decision is traceable." The homepage lists a price of 0 USD.
The practical meaning of the license is worth stating plainly. AGPL-3.0 means the source is open, but if you run a modified Routerly as a network service for others, you must share your changes. For a team running it internally as its own gateway, that obligation rarely bites — but read the license before you white-label it.
What are the honest caveats before you self-host?
No independent review, tutorial, or benchmark of Routerly could be located as of this writing — despite multiple searches, the only "coverage" beyond the vendor's own pages is the maintainer's own Medium comment. Everything substantive in this guide comes from Inebrio's Docker Hub listing, GitHub repository, documentation, and blog, so treat every performance claim as vendor-published, not independently verified.
The vendor's own benchmark post — 91.7% on HumanEval with its LLM policy versus 93.3% for what the vendor calls "Opus 4.8", at 37% of the cost — is explicitly labeled "Preliminary, small-sample comparison" by the vendor itself, with 60 questions per model and cost figures the vendor says to "treat as current API pricing rather than a controlled simultaneous run." Take it as a marketing data point, not a measurement. Also: the vendor itself admits it does not yet measure the extra latency its own hop adds, so no latency claim appears in this guide.
One versioning caution: GitHub's releases page showed v0.6.0 published on 2026-11-04, but the vendor's own install one-liners and docs still pin v0.5.3, with the vendor noting the default channel resolves to an older release. Do not assume "latest" and "v0.6.0" are the same thing right now; check which tag your install actually pulls.
Sources
All sources below were fetched and read for this guide in early November 2026; every claim attributed to them is vendor-published, and no independent third-party coverage of Routerly was found.
- Inebrio — Routerly on Docker Hub (https://hub.docker.com/r/inebrio/routerly): vendor one-pager with tagline, feature list, quick-start commands, provider table, and environment variables.
- Inebrio/Routerly on GitHub (https://github.com/Inebrio/Routerly): public repository — README quick start, routing-policy table, provider list, comparison table, dashboard and CLI docs; license listed as AGPL-3.0.
- Inebrio/Routerly releases (https://github.com/Inebrio/Routerly/releases): release history including v0.6.0 dated 2026-11-04 and v0.4.0 dated 2026-08-06.
- Routerly vendor blog (https://blog.routerly.ai/routerly-vs-fable-5-opus-4-8, published 2026-06-10): HumanEval and MMLU benchmark post with method notes, including the "Preliminary, small-sample comparison" wording.
- Routerly homepage (https://www.routerly.ai/): product page — tagline, policy count, provider footer, listed price of 0 USD.
- Routerly documentation (https://doc.routerly.ai/): feature reference — the eleven routing policies, profiles, token optimizers, permissions, notifications, and MCP server.
- Routerly docs — Budgets and Limits (https://doc.routerly.ai/concepts/budgets-and-limits): the Global to Router to per-token hierarchy, spend groups, window types, and the budget_exceeded 503 behavior.
- Carlo Satta, maintainer — Medium comment (2026-11-04): "I maintain Routerly (self-hosted, AGPL, no paid tier), where routing is a set of composable policies and each decision is traceable."
Frequently Asked Questions
Is Routerly the same thing as LiteLLM?
No. Routerly is a self-hosted gateway product with a dashboard, budget enforcement, and user management; LiteLLM is an open-source Python library for calling many models from your own code. You might use LiteLLM inside your application and still point it at Routerly as the gateway.
Is Routerly the same as OpenRouter?
No. OpenRouter is a hosted API proxy: you pay them and they route to providers. Routerly is software you run yourself, on your own infrastructure, with no database and your own API keys. Choose OpenRouter when you want zero infrastructure; choose Routerly when you want the gateway, the logs, and the credentials on your side of the wall.
Does Routerly work with my existing AI tools?
Almost certainly yes, if they speak the OpenAI or Anthropic API formats. The vendor lists Cursor, Continue.dev, Open WebUI, LibreChat, LangChain, LlamaIndex, and plain curl as working without modification — you change only the base URL, plus an API key the gateway issues.
What happens when a Routerly budget is exhausted?
The gateway returns an HTTP 503 error with a budget_exceeded message, and no provider call is made. Because limits are enforced before the provider is called, an exhausted budget cannot silently turn into a surprise bill.
Is there really no database?
Correct, per the vendor: configuration and usage data live in plain JSON files under the data directory, with no PostgreSQL, no Redis, and no migrations. Backups are file copies — which is also why you should make sure that directory is itself backed up and access-controlled.
