Routing
LLM model routing with Jev: a practical example
An independent guide by Jev AI · Reviewed · 阅读中文版
An LLM router chooses which approved model should handle a task. Jev can make that bounded selection from task context, candidate descriptions and constraints, while your application performs the eventual model call. The router does not discover models or verify their capabilities for you.
This example compares a hypothetical fast 32k-context candidate with a hypothetical reasoning candidate described as supporting 200k context and tools. Those are example labels and assumptions, not capability claims for named commercial models.
What you will learn
- Filter hard constraints before asking for semantic judgment.
- Provide accurate candidate descriptions and an escalation branch.
- Compare total accepted-task cost with a fixed-model baseline.
On this page
- Where the routing decision belongs
- Describe candidates without asking Jev to guess
- Treat escalation as a useful outcome
- Evaluate routing against a fixed-model baseline
- Request and code examples
- Interpret the result
- Troubleshoot a failed request
- Checks before integration
- Common questions
- Sources and next steps
Where the routing decision belongs
Place routing after you know the task and available models, but before launching the expensive downstream call. Use code to remove unavailable models, models that lack required permissions and candidates whose known context limit cannot accommodate the request. These constraints should not be inferred from a model name.
If the remaining decision is already an exact rule, keep it as code. A semantic router is useful when the task description, ambiguity or quality needs matter. It adds another call, so measure whether the selection benefit exceeds that added latency and cost.
- Gather task requirements and an approved candidate list.
- Enforce exact constraints and evaluate the remaining semantic choice.
- Validate the returned candidate ID; escalate or call the selected model.
Describe candidates without asking Jev to guess
The model-route preset expects task and a nonempty candidates array. Each candidate needs a unique id. Add capability descriptions, approximate cost and latency information that you can maintain. The preset reserves escalate as a fallback and turns candidate IDs into Choice criteria.
priorities, constraints and stakes explain the selection context. A refund dispute with a long record and tool requirements has different needs from a short classification job. Keep the input small enough for the routing model itself: describing a 100k-context task does not mean sending the entire downstream transcript to the router.
| Input | Why it matters |
|---|---|
| task | Describe the work and quality requirements |
| candidates | Unique IDs and known capabilities |
| priorities | Relative importance of quality, context, tools, cost or latency |
| constraints | Boundaries that remain enforced in application code |
Treat escalation as a useful outcome
Read answers.model.choice and verify that it is one of your approved IDs or escalate. A missing field, malformed distribution, no adequate candidate or ambiguous result should lead to a defined review path. Never take an arbitrary model ID from response text and forward it to another API.
If you add an acceptance threshold, choose it from labeled routing cases rather than copying a number from a tutorial. Evaluate the runner-up probability and the types of mistakes that matter. A cheap but incapable model can cost more once retries and human correction are counted.
Illustrative example, not a live API result
// Application sketch after validating the native response:
const approved = new Set(["fast-model", "reasoning-model"]);
const choice = result.answers.model.choice;
if (choice === "escalate" || !approved.has(choice)) {
// Queue for a configured fallback or human review.
} else {
// Recheck availability and map this ID to your configured provider model.
}Evaluate routing against a fixed-model baseline
Create a small labeled workload containing simple tasks, long-context tasks, tool-dependent tasks and tasks with insufficient information. Run the same downstream evaluation with a fixed model and with routing. Judge task completion quality rather than whether the router selected the label you expected in one example.
Track routing latency, downstream latency, accepted-task quality, retries, fallback frequency and total provider cost. Keep a held-out set when tuning descriptions or thresholds. Re-evaluate when candidate capabilities, model versions or traffic mix change; an earlier routing result does not describe a permanently optimal choice.
Request and code examples
Create an API key in OmniaKey and set OMNIAKEY_API_KEY in your terminal or server environment. The curl example requires curl 7.76 or newer for --fail-with-body; the JavaScript example requires Node.js 20 or newer. Use synthetic inputs. Requests use your account balance; check OmniaKey for current pricing and availability.
Save the JavaScript as jev-example.mjs, then run node jev-example.mjs. Keep the key out of frontend code and Git. Example inputs stay in English to match the playground presets; evaluate other input languages separately.
curl
curl --fail-with-body --max-time 25 'https://api.omniakey.com/v1/alpha/search' \
-H "Authorization: Bearer ${OMNIAKEY_API_KEY}" \
-H 'Content-Type: application/json' \
--data '{
"model": "jev-latest",
"state": {
"task": "Review a complex customer dispute with 100k context and tool use",
"candidates": [
{
"id": "fast-model",
"description": "Fast general model with 32k context",
"cost": "low",
"latency": "low"
},
{
"id": "reasoning-model",
"description": "Strong reasoning, 200k context, and tool use",
"cost": "high",
"latency": "medium"
}
],
"priorities": [
"quality",
"context",
"tool_use",
"cost"
],
"constraints": [
"Customer data must remain within approved tools"
],
"stakes": "high"
},
"questions": {
"model": {
"type": "choice",
"instructions": "Select the best approved candidate using the stated quality, context, tools, cost, latency, and stakes. Do not assume undocumented capabilities. Choose escalate when no candidate is adequate or evidence is insufficient.",
"criteria": {
"fast-model": "Choose this approved model if its documented capabilities satisfy the task, priorities, constraints, and stakes: {\"id\":\"fast-model\",\"description\":\"Fast general model with 32k context\",\"cost\":\"low\",\"latency\":\"low\"}",
"reasoning-model": "Choose this approved model if its documented capabilities satisfy the task, priorities, constraints, and stakes: {\"id\":\"reasoning-model\",\"description\":\"Strong reasoning, 200k context, and tool use\",\"cost\":\"high\",\"latency\":\"medium\"}",
"escalate": "No approved candidate can safely satisfy the task, or the provided evidence is insufficient to select one."
}
}
}
}'Show the complete Node.js example
Node.js 20+
// Node.js 20+. Save as jev-example.mjs and run: node jev-example.mjs
// Set OMNIAKEY_API_KEY in the server/terminal environment first.
const key = process.env.OMNIAKEY_API_KEY;
if (!key) throw new Error("Set OMNIAKEY_API_KEY before running.");
const body = {
"model": "jev-latest",
"state": {
"task": "Review a complex customer dispute with 100k context and tool use",
"candidates": [
{
"id": "fast-model",
"description": "Fast general model with 32k context",
"cost": "low",
"latency": "low"
},
{
"id": "reasoning-model",
"description": "Strong reasoning, 200k context, and tool use",
"cost": "high",
"latency": "medium"
}
],
"priorities": [
"quality",
"context",
"tool_use",
"cost"
],
"constraints": [
"Customer data must remain within approved tools"
],
"stakes": "high"
},
"questions": {
"model": {
"type": "choice",
"instructions": "Select the best approved candidate using the stated quality, context, tools, cost, latency, and stakes. Do not assume undocumented capabilities. Choose escalate when no candidate is adequate or evidence is insufficient.",
"criteria": {
"fast-model": "Choose this approved model if its documented capabilities satisfy the task, priorities, constraints, and stakes: {\"id\":\"fast-model\",\"description\":\"Fast general model with 32k context\",\"cost\":\"low\",\"latency\":\"low\"}",
"reasoning-model": "Choose this approved model if its documented capabilities satisfy the task, priorities, constraints, and stakes: {\"id\":\"reasoning-model\",\"description\":\"Strong reasoning, 200k context, and tool use\",\"cost\":\"high\",\"latency\":\"medium\"}",
"escalate": "No approved candidate can safely satisfy the task, or the provided evidence is insufficient to select one."
}
}
}
};
const response = await fetch("https://api.omniakey.com/v1/alpha/search", {
method: "POST",
headers: {
Authorization: `Bearer ${key}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
signal: AbortSignal.timeout(25_000),
});
if (!response.ok) {
const retryAfter = response.headers.get("retry-after");
throw new Error(`Jev HTTP ${response.status}; Retry-After: ${retryAfter ?? "not supplied"}`);
}
const result = await response.json();
console.log(JSON.stringify(result, null, 2));
// Inspect and validate the answer before connecting it to any external action.
The JSON below is the native API body. This guide's playground link selects the matching workflow preset; preset input and native request are different shapes. To use Custom Jev instead, paste this native JSON.
Copy native JSON for the playground
Native JSON
{
"model": "jev-latest",
"state": {
"task": "Review a complex customer dispute with 100k context and tool use",
"candidates": [
{
"id": "fast-model",
"description": "Fast general model with 32k context",
"cost": "low",
"latency": "low"
},
{
"id": "reasoning-model",
"description": "Strong reasoning, 200k context, and tool use",
"cost": "high",
"latency": "medium"
}
],
"priorities": [
"quality",
"context",
"tool_use",
"cost"
],
"constraints": [
"Customer data must remain within approved tools"
],
"stakes": "high"
},
"questions": {
"model": {
"type": "choice",
"instructions": "Select the best approved candidate using the stated quality, context, tools, cost, latency, and stakes. Do not assume undocumented capabilities. Choose escalate when no candidate is adequate or evidence is insufficient.",
"criteria": {
"fast-model": "Choose this approved model if its documented capabilities satisfy the task, priorities, constraints, and stakes: {\"id\":\"fast-model\",\"description\":\"Fast general model with 32k context\",\"cost\":\"low\",\"latency\":\"low\"}",
"reasoning-model": "Choose this approved model if its documented capabilities satisfy the task, priorities, constraints, and stakes: {\"id\":\"reasoning-model\",\"description\":\"Strong reasoning, 200k context, and tool use\",\"cost\":\"high\",\"latency\":\"medium\"}",
"escalate": "No approved candidate can safely satisfy the task, or the provided evidence is insufficient to select one."
}
}
}
}Interpret the result
For the supplied synthetic dispute, the reasoning candidate is described as meeting the larger context requirement, but the guide does not assert an actual inference result. Run the example to see which option your available Jev version returns and compare its distribution with your requirements.
The native response uses answers.model for the question, while the top-level model is the Jev model version that performed routing. They are different concepts. usage is the routing request's token usage, not the bill for the selected downstream model.
Troubleshoot a failed request
- Invalid request (for example, 400)
- Check model, state, questions and each question's type, options or range. Do not send a chat messages payload to this endpoint.
- Authentication or account rejection (such as 401, 402 or 403)
- Check the key, balance, model access and current provider policy. A status alone does not uniquely identify the cause. Inspect your OmniaKey account without pasting credentials into support text.
- Rate limit (429)
- Read Retry-After if supplied, reduce concurrency and use bounded backoff. These examples do not retry automatically; duplicate requests may add cost.
- 5xx, network failure or timeout
- Check service status and connectivity before deciding to retry. The 25-second timeout ends the client's wait; it does not guarantee upstream cancellation or avoid billing.
- HTTP success with an unusable answer
- Validate the response shape, question names and answer types. Missing answers or unknown options need a fallback or review, rather than coercion followed by execution.
Checks before integration
- Remove the long-context candidate: the router should have an escalation route instead of inventing one.
- Make all candidate capabilities unknown and check how uncertainty is handled.
- Supply duplicate candidate IDs: correct the input before making a request.
- Change task language and complexity independently to detect brittle routing.
- Include cases where a fixed deterministic rule is sufficient and routing adds no value.
Common questions
Does Jev call the selected model?
No. This preset returns a decision. Your application validates it, maps the approved ID to a real configured model and performs the downstream call with its own credentials and limits.
Will routing automatically reduce my API bill?
No savings are established by this example. Include the router's cost, retries, quality failures and downstream usage when comparing against a fixed-model baseline.
Sources and next steps
Protocol details were checked against OmniaKey's Jev guide and authentication documentation. Request examples pass local schema checks; this article does not claim a live performance evaluation of each workflow. This site is independent of the model developer and OmniaKey.
Try it with your own task
Start with a synthetic input and inspect the native request and model answer. Open OmniaKey key management with your account, or follow its sign-up flow if you are new.