The local model conversation gets noisy when it turns into a belief system: open-weight versus closed, local versus hosted, frontier versus small.
Those debates can miss the engineering question that matters first: which part of the workload deserves which cost, latency, quality, privacy, and control profile?
Teams can audit that decision route by route.
Model routing is where platform strategy meets FinOps. Without route-level cost, quality, and fallback data, leaders are funding preferences before they have operating decisions.
A serious model strategy should start by decomposing the workflow. Some steps need frontier reasoning. Some need fast classification. Some need retrieval. Some need summarization. Some need code search. Some need deterministic business logic and should not be model calls at all.
Once the workflow is separated, local and open-weight models become architectural options, not ideology.
Where local models fit
Local and open-weight models can be useful for bounded, high-volume, privacy-sensitive, latency-sensitive, or cost-sensitive steps.
That does not mean they are automatically cheaper or better.
Running a model locally can reduce some marginal inference costs. It can also shift cost into hardware, hosting, model operations, routing logic, evals, fallback design, monitoring, and quality control. Someone still pays for the system. The bill just moves.
Consider an AI coding assistant.
It might use a local code index to find relevant files. It might use a smaller local model to classify the type of change, summarize nearby code, or rank candidate files. It might route a risky refactor or unclear design question to a frontier model. It might block file edits until tests and eval gates pass.
That is different from saying “run everything locally.”
The value comes from routing the right step to the right capability.
The same pattern applies outside coding. A claims workflow might use a small model to classify document type, a retrieval system to find the right policy, a frontier model to reason over an unusual exception, and deterministic rules to decide whether the action needs approval. Each step has a different quality bar. Treating them as one model decision hides that.
The routing view also helps finance and risk teams participate. They do not need to debate model benchmarks in the abstract. They can inspect which steps send data outside a boundary, which routes have predictable volume, which routes need low latency, and which routes should fail closed.
Build the routing table
A practical routing table does not need to start fancy. It needs to be explicit.
For each step in the workflow, write down:
- Task type.
- Input sensitivity.
- Required quality bar.
- Latency budget.
- Cost budget.
- Model route.
- Fallback route.
- Eval gate.
- Human review threshold.
The table replaces a general debate about whether local models are good enough with a measurable question: is this model good enough for this step under these constraints?
It also exposes where a model call should disappear. Some routes should send the task to search, a rules engine, a database query, or a human reviewer. If the approval threshold is a rule, route to the rule. A cheaper model is still waste when the task never needed a model.
One routing row might be: classify incoming support ticket, low sensitivity, local model, confidence threshold, fallback to hosted model, human review when refund intent appears.
Another local route might be acceptable for repository navigation if the eval shows it finds the right file set often enough and never edits files directly. A frontier route might be required for cross-service design changes. A human review gate might be required when the route touches production configuration or customer-visible copy.
The routing table should also include data movement. If the step involves sensitive customer data, regulated documents, proprietary code, or contract terms, cost is only one part of the decision. The team also needs to know where the data goes, who can inspect it, how long it is retained, and which controls apply.
Measure routes, not preferences
The easiest way to make this debate practical is to attach evals to routes.
Each route should have acceptance criteria. Does the local model classify the request correctly? Does it retrieve the right files? Does it preserve required facts in a summary? Does it stay inside latency and cost budgets? Does the fallback catch uncertain cases?
If the answer is unknown, the route is an opinion.
Capability pages and deployment docs are starting points. They are not proof that a particular route works for your workload. The only useful proof is workload-specific evaluation.
Those evals should include fallback behavior. If the local model is uncertain, does the workflow escalate to a stronger model, ask a human, or stop? If the fallback is too loose, the team may save money on easy cases and lose control on hard ones.
Token budgets belong in the same review. A workflow can quietly become expensive because it sends every step to the most capable model, retrieves too much context, retries without a stop condition, or summarizes material that a search index could have narrowed first. Routing is how those costs become visible.
Local and open-weight models belong where the workload is bounded, measurable, and paired with a fallback route.
Leaders need a routing model that makes cost and control visible.
Before declaring a model strategy, ask which model calls in your system actually require frontier reasoning, and which are expensive habit. Here are five ways to make that question concrete.
A few things to try
- Decompose one workflow. List each step from intake to final action, then mark whether it needs generation, retrieval, search, a rule, or human judgment. Remove model calls from steps with a deterministic answer.
- Build the first routing table. For every remaining model step, record the input sensitivity, quality threshold, latency budget, cost budget, primary route, and fallback. A blank field is a decision the team has not made yet.
- Evaluate one bounded route. Choose a high-volume task such as classification, summarization, or repository navigation. Test a local or smaller model against real examples, including difficult cases, and compare quality, latency, and total operating cost with the current route.
- Define uncertainty behavior. Decide when the route should retry, escalate to a stronger model, ask for human review, or stop. Test the fallback with the same care as the primary model; hard cases are where loose routing becomes expensive.
- Review data movement and ownership. For each route, show where inputs go, how long they are retained, who operates the model, and who owns failures. Include hardware, hosting, monitoring, evals, and support when comparing costs.
Which model calls in your system actually require frontier reasoning, and which are expensive habit?