What is the difference between AI routing and LLM routing?

LLM routing chooses between language models. AI routing is the broader decision that also includes agents, tools, skills, workflows and people as targets, under one policy.

Where does the routing layer sit?

Between the applications and the models, agents and tools they use - usually as an OpenAI-compatible endpoint, so existing clients move by changing one base URL.

Does AI routing add latency?

The decision itself takes microseconds to a few milliseconds: signals are hashed features and the scoring is a handful of small models. The model call that follows dominates, and routing to a smaller model usually makes the whole request faster.

Glossary

AI routing

AI routing is the practice of deciding, per request, which AI capability should handle it - a model, an agent, a tool, a skill, a workflow or a person - under the caller's policy, with a record of why.

Language models are one kind of thing a request can be sent to. A coding task may be better done by an agent that can run tests; a lookup by a tool; a translation by a skill with its own instructions; a complaint by a person. AI routing is the general form of the decision an LLM router makes: given a request, which of the capabilities the organisation owns should handle it, and under what rules.

Seven kinds of routing fall out of that definition. Model routing picks between language models. Agent routing hands a request to an autonomous worker when it needs doing rather than answering. Tool routing sends it to a function, an MCP tool or a search. Skill routing picks a packaged set of instructions. Workflow routing chooses a plan - a persona, then a skill, then a model - executed step by step. Human routing puts a person in the path when judgement, consent or escalation is required. Policy routing runs first and bounds all the others: data boundary, region, tenant, budget and allowed targets decide what may be a candidate at all.

A routing layer belongs between the applications and the capabilities, where it can see every request and apply one policy. In practice that is an OpenAI-compatible endpoint the applications already know how to call, with the catalogue of targets, the rules and the learned state behind it. The applications stop naming models; the catalogue does.

What makes routing trustworthy is the trace. A decision that cannot be explained cannot be audited, improved or defended to a regulator. Every routed request should record the candidates, their scores and prices, the constraints that removed some of them, the confidence of the choice and, later, the outcome - so the organisation learns from its own traffic instead of from a vendor's benchmark.

On this site

AI routing: the short answers

What is the difference between AI routing and LLM routing?: LLM routing chooses between language models. AI routing is the broader decision that also includes agents, tools, skills, workflows and people as targets, under one policy.
Where does the routing layer sit?: Between the applications and the models, agents and tools they use - usually as an OpenAI-compatible endpoint, so existing clients move by changing one base URL.
Does AI routing add latency?: The decision itself takes microseconds to a few milliseconds: signals are hashed features and the scoring is a handful of small models. The model call that follows dominates, and routing to a smaller model usually makes the whole request faster.

See the definition at work on a request of your own.

OpenSmartRoute is open source and the free plan keeps the full trace of every decision. Type a request in the playground and read the ranked candidates.

Create a free account Every term

Free plan, no card. Fifteen thousand decisions a month with the full trace.