As Generative AI systems move from prototype to production, LLM inference costs can quickly become the single largest line item in your cloud budget. A pipeline processing 10 million tokens a day through flagship frontier models like Claude 3.7 Sonnet or GPT-4o can cost thousands of dollars per month if unoptimized.
However, by leveraging Prompt Caching, Tiered Model Routing, Structured Token Budgeting, and Batch Processing, engineering teams routinely reduce their API bills by 60% to 85% with zero degradation in output quality.
This guide provides practical architectural patterns, code examples, and mathematical models to optimize your LLM unit economics.
1. The Economics of LLM APIs: Input vs Output Tokens
LLM APIs charge on an asymmetric pricing model where output tokens are 3x to 5x more expensive than input tokens:
| Model Tier | Representative Models | Input Cost (per 1M tokens) | Output Cost (per 1M tokens) | Best For |
|---|---|---|---|---|
| Frontier / Flagship | Claude 3.7 Sonnet, GPT-4o | $2.50 – $3.00 | $10.00 – $15.00 | Complex reasoning, code generation, synthesis |
| Fast / Lightweight | Claude 3.5 Haiku, GPT-4o-mini, Gemini 2.0 Flash | $0.10 – $0.80 | $0.40 – $3.20 | Classification, extraction, summarization, routing |
| Open Weights / Self-Hosted | DeepSeek V3, Llama 3.3 70B (via Groq/Together) | $0.15 – $0.60 | $0.60 – $1.20 | High-throughput background transformations |
The Golden Rule of Token ROI
Every prompt optimization strategy revolves around two levers:
- Minimize Output Tokens: Keep output formats strictly concise (e.g., JSON schemas without verbose prose).
- Cache or Route Input Tokens: Ensure large system prompts and RAG contexts are charged at cached or lightweight rates.
2. Prompt Caching: The 90% Discount Mechanism
Modern frontier providers support prefix caching, allowing models to reuse pre-computed Key-Value (KV) attention states across repeated requests:
- Anthropic Claude: Up to 90% discount on cached prompt tokens, with a 5-minute TTL refreshed on every hit.
- OpenAI: Automatic prompt caching gives a 50% discount on prompt prefixes over 1,024 tokens.
- Google Gemini: Explicit context caching with discounts up to 75% on input token costs.
Implementing Anthropic Prompt Caching
To enable caching in Anthropic API calls, structure your static context (e.g., system instructions, large documentation chunks, tool schemas) with cache_control breakpoints:
import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();
const response = await anthropic.messages.create({
model: 'claude-3-7-sonnet-20250219',
max_tokens: 1024,
system: [
{
type: 'text',
text: 'You are an enterprise support specialist with complete access to the company knowledge base.',
},
{
type: 'text',
text: LARGE_KNOWLEDGE_BASE_DOCUMENTATION, // 15,000 tokens
cache_control: { type: 'ephemeral' }, // Caches this prefix
},
],
messages: [
{
role: 'user',
content: 'How do I configure SAML 2.0 SSO with Okta?',
},
],
});
console.log('Cache creation tokens:', response.usage.cache_creation_input_tokens);
console.log('Cache read tokens:', response.usage.cache_read_input_tokens); // Billed at 90% off!
3. Tiered Model Cascading & Router Architecture
Never send every user query to your most expensive model. In practice, 70%+ of incoming tasks are simple intent classifications, basic queries, or formatting requests that a fast model handles with parity.
[ User Request ]
│
▼
[ Fast Classifier: GPT-4o-mini / Haiku ]
│
┌─────────────┴─────────────┐
▼ ▼
[ Confidence >= 0.9 ] [ Complex / Low Confidence ]
│ │
▼ ▼
[ Fast Model Executes ] [ Frontier Model: Claude 3.7 / GPT-4o ]
Cascading Pattern Implementation
async function executeOptimizedQuery(userPrompt: string) {
// Step 1: Low-cost triage (< $0.0001)
const complexityEvaluation = await quickClassify(userPrompt);
if (complexityEvaluation.isSimple) {
return await callModel('gpt-4o-mini', userPrompt);
}
// Step 2: Escalate only difficult queries to frontier models
return await callModel('claude-3-7-sonnet-20250219', userPrompt);
}
4. Prompt Engineering for Token Efficiency
A. Pruning JSON Schemas and Field Names
Verbose JSON key names repeated thousands of times consume unnecessary context.
// BAD: Verbose payload consumes 42 tokens
{
"customer_account_identification_number": "ACC-9921",
"total_aggregated_annual_recurring_revenue": 142000.50
}
// GOOD: Concise schema consumes 18 tokens (57% savings)
{
"accountId": "ACC-9921",
"arr": 142000.50
}
B. Enforcing Structured Outputs Without Fluff
Instruct models to output strictly raw JSON without conversational preambles ("Here is the requested information..."):
const prompt = `Return the analysis strictly as valid JSON adhering to the schema. No prose, no markdown fences, no conversational preamble.`;
5. Non-Real-Time Workloads: 50% Off via Batch APIs
For offline data enrichment, overnight vector embeddings, and synthetic dataset generation, use provider Batch APIs (OpenAI Batch API, Anthropic Message Batches):
- 50% Flat Discount: Both input and output tokens are billed at exactly half price.
- Higher Rate Limits: Separate, significantly higher queries-per-minute (QPM) ceilings.
- Turnaround Window: Completed asynchronously within 24 hours (usually < 30 minutes).
6. Measuring Your Blended Cost-Per-Query (CPQ)
Track your token unit economics with the standard Cost Per Query (CPQ) formula:
$$\text{CPQ} = \frac{(\text{Cached Input} \times P_{\text{cache}}) + (\text{Uncached Input} \times P_{\text{input}}) + (\text{Output Tokens} \times P_{\text{output}})}{1,000,000}$$
Use the AI Cost Calculator to model provider pricing comparisons and test your token footprint with the AI Token Counter.