Reduce LLM inference cost by matching each task to an evaluated model route, batching latency-tolerant work and reusing cached material only when its scope and meaning remain valid. Count retries, fallbacks, cache operations and review effort. A cheaper first model call is not necessarily a cheaper completed task.

This guide provides routing criteria, a runnable token-cost calculation and a benchmark worksheet. All example rates and workload assumptions are fictional. The arithmetic is locally checked; no live model quality, latency or savings benchmark is claimed.

Measure the full request path before optimizing

Record the selected model configuration, input and output usage, retries, fallback calls and relevant cache behavior. Add fixed serving or platform charges and engineering operations where they apply. Use the provider’s documented usage dimensions; not every model or deployment bills identically.

Track cost per request and per accepted task. Define acceptance with the application owner: correct result, required evidence, valid output contract and permitted action. An apparent token saving can disappear when users retry failed tasks or specialists repair them manually.

Microsoft’s AI cost guidance discusses caching, batching, routing and model selection as separate levers. Treat each as a testable change rather than applying all of them together and losing the ability to explain the result.

Route by task requirements and measured capability

Start with explicit task classes and a small approved model set. A routine extraction or classification task may work with a lower-cost model; an ambiguous multi-document task may need another route. Validate that distinction on representative held-out cases.

Do not trust a model’s self-reported confidence as the sole routing criterion. Use validated signals and record why the route was chosen. Keep a bounded fallback path when output checks fail, and distinguish recoverable quality failures from authorization denials.

Fallback calls are additional work

If a small model runs first and a larger model retries the task, both calls may be billed. A high fallback rate can erase the expected saving and add latency. Measure the distribution by task class so a weak route does not stay enabled simply because its unit price is low.

Separate asynchronous batching from request packing

A provider’s supported asynchronous batch API and putting multiple unrelated user tasks into one prompt are different techniques. The former generally manages independent requests through a batch job; the latter changes context isolation and output parsing.

For offline workloads, confirm the current provider’s completion window, supported models, pricing and limits. Preserve a unique request identifier, detect missing results and retry only failed items. Do not assume output order equals input order unless the documented contract guarantees it.

Avoid combining incompatible tenants or sensitive tasks into a shared prompt for convenience. If request packing is used at all, test isolation and result mapping carefully. Include the extra parsing and failure recovery in the cost model.

Prompt caching and semantic answer caching protect different things

Prompt caching

Prompt caching reuses eligible repeated input processing according to the provider’s rules. It does not mean the final answer is reused or that generation disappears. Claude’s prompt-caching documentation describes prefix-based reuse and configurable cache behavior. Check the selected provider and model rather than assuming identical cache semantics everywhere.

Keep stable instructions and reusable material stable when the API permits it, but never preserve obsolete policy just to improve cache hits. Inspect actual cache usage and applicable read/write charges.

Semantic answer caching

A semantic cache may reuse an answer for a sufficiently similar request. Similar wording can hide a different customer, date, quantity, jurisdiction or permission scope. Similarity is a retrieval signal, not proof of answer equivalence.

The GenCache research paper discusses the difficulty of reusing responses when prompts have meaningful variations. Its experimental results concern that system and workload; do not transfer its reported improvements to an untested application.

Authorize the cached sources for the current caller, check source and policy freshness, and preserve task-specific values. A question such as “What is my order status?” is not safely reusable across users merely because the embedding is similar. Start with narrowly defined, low-risk cache eligibility and measure incorrect hits as well as hit rate.

Runnable token-cost calculator with explicit assumptions

The following script assumes 100,000 monthly tasks, 70,000 small-model calls, 30,000 direct larger-model calls and a 10% fallback rate among the small-model tasks. Each call uses 2,000 input and 300 output tokens. Rates are fictional currency units per million tokens: small 0.5 input / 2 output; large 2 input / 8 output.

"""Fictional token-cost arithmetic, not model performance or provider pricing."""
def token_cost(requests,input_tokens,output_tokens,input_rate,output_rate):
    return requests*(input_tokens*input_rate+output_tokens*output_rate)/1_000_000

requests=100_000
small_requests=70_000
direct_large=requests-small_requests
fallbacks=small_requests*0.10
baseline=token_cost(requests,2000,300,2,8)
small=token_cost(small_requests,2000,300,0.5,2)
large=token_cost(direct_large+fallbacks,2000,300,2,8)
routed=small+large
assert baseline==640
assert abs(routed-348.8)<1e-8
print({'all_large':baseline,'small_calls':small_requests,
       'large_calls_including_fallback':direct_large+fallbacks,
       'routed_tokens_only':routed})

The local calculation yields 640 fictional currency units for the all-large path and 348.8 for the routed token-only path, including 7,000 extra larger-model fallback calls. These are declared-input arithmetic results, not measured cost savings. No claim is made that either model route meets a real quality target.

Add router processing, cache cost, platform charges, review, retries and any provider-specific billing dimensions. If fallback calls have longer prompts or outputs, model them separately. The simple example deliberately keeps token counts equal so the calculation is easy to inspect.

A benchmark worksheet for your own application

This is a measurement template, not a filled performance benchmark. Run comparable workload conditions, disclose sample counts and preserve difficult slices. Use the LLM testing playbook to structure the quality and boundary checks.

Roll out optimization as a behavior change

1. Establish a measured baseline and task acceptance criteria.
2. Choose one optimization hypothesis and define its affected task class.
3. Evaluate quality, access, latency and full cost.
4. Inspect false cache hits and fallback-heavy routes.
5. Deploy gradually with observable stop conditions.
6. Keep a rollback configuration and an accountable operating owner.

For retrieval-specific costs, use the separate RAG cost guide and calculator. Keep that broader indexing and retrieval budget separate from this article’s inference-routing arithmetic.

Common questions

Optimize the complete task, not only the first call

For related strategies, read RAG versus fine-tuning and prompt engineering. To assess model routing, evaluation and operating cost for your workload, discuss your production AI reliability roadmap.