What is an LLM router?
An LLM router is a component that decides which language model should answer each incoming request. It sits between your application and a set of models. It reads the request, picks a target, forwards it, and hands back the answer.
Three approaches to that decision are common. A rule-based router matches on task type, or on a label the application already carries. A classifier-based router scores how hard the prompt looks, usually with a small model, and picks accordingly. A cascading router tries the cheap model first and escalates only when the answer fails a check. Most routers in production combine a thin rule layer with one of the other two.
Routing usually lives inside an AI gateway, next to authentication, rate limiting, and logging. It can also run as a service of its own, or as a library inside the application.
In plain words
Think of the triage nurse in a busy emergency room. Nobody sends every patient straight to the surgeon. The nurse looks at each case, handles the scraped knee on the spot, and walks the chest pain to a specialist. An LLM router does that same triage for your prompts: a quick look, then the cheapest option that still does the job properly.
Why it matters
- Cost. Most requests in a real product are easy. Sending all of them to the largest model means paying a premium for work a small model would finish correctly.
- Speed. A smaller model answers sooner. That gap shows up most in chat, and anywhere a person is waiting.
- Freedom to swap. Your application talks to the router, not to a provider SDK. Adding a model, or replacing one, turns into a config change.
- Failover. When a provider is rate-limited or down, the router can send the same request elsewhere instead of failing.
Common pitfalls
- The router costs more than it saves. A classifier that adds 300 ms and its own model call can eat the whole benefit. Measure the routed path end to end.
- Quality drops where you do not look. An aggregate score hides one badly routed segment. Evaluate per route, not only overall.
- Routing on a weak signal. Prompt length is not difficulty. A short question can need deep reasoning, and a long paste of logs can need almost none.
- Unlogged model switches. If you cannot tell which model answered, you cannot reproduce a bug report. Log the routing decision with every response.
- Confusing a router with a gateway. A gateway is the single front door that handles auth, limits, and logging for everything behind it. A router picks the model. One product often does both, yet they remain two different jobs.
Related articles:
- What is an LLM? - The kind of model a router chooses between.
- What is an API gateway? - The pattern an AI gateway is built on.
- What is AI inference? - What happens once the chosen model gets the request.
- What is LLM observability? - How you find out whether a routing decision was any good.
- What is a token in AI? - The unit the cost difference gets measured in.
Want to stay one step ahead?
Don't miss our best insights. No spam, just practical analyses, invitations to exclusive events, and podcast summaries delivered straight to your inbox.