LLM Token Counting & Context Window Budgeting: BPE Tokenizers & Cost Optimization

Large Language Models (LLMs) do not read text character-by-character or word-by-word; they process text as numerical sequences called tokens.

Token consumption governs everything from API billing (where pricing is typically calculated per 1 million input and output tokens) to latency (Time to First Token vs Time Per Output Token) and context window limits (where exceeding maximum token length causes abrupt prompt truncation or 400 context_length_exceeded errors).

This guide explores the mechanics of Byte-Pair Encoding (BPE) tokenizers, details how to budget tokens for complex multi-turn agent pipelines, explains prompt caching economics, and provides programmatic code snippets for accurate token counting.

---

1. How Byte-Pair Encoding (BPE) Works

BPE is a subword tokenization algorithm that iteratively replaces the most frequent pairs of bytes or characters in a corpus with a single new token.

• Common English words typically map to 1 token (e.g. "developer" = 1 token).
• Rare, technical, or non-English words are split into multiple subword tokens (e.g. "Subnetting" = ["Sub", "net", "ting"] = 3 tokens).
• Code and Whitespace: Spaces and indentation are significant. In Python or YAML, four leading spaces might consume 1 or 2 tokens depending on the vocabulary.

Text: "Authentication failed for user: admin"
Tokens (cl100k_base): ["Authent", "ication", " failed", " for", " user", ":", " admin"]
Total: 7 tokens (~38 characters ≈ 5.4 chars/token)

Vocabulary Comparison Across Major Models

Model Family	Tokenizer Encoding	Vocab Size	Efficiency on Code / Non-Latin
GPT-4 / GPT-3.5	cl100k_base	~100,000 tokens	Baseline standard
GPT-4o / GPT-4o-mini	o200k_base	~200,000 tokens	~15–20% fewer tokens on code and non-English
Claude 3 / 3.5 / 3.7	Anthropic BPE	~65,000 tokens	Highly compact on natural prose & XML
Llama 3 / DeepSeek V3	Tiktoken-style BPE	~128,000 tokens	Optimized for multilingual and reasoning traces

---

2. The Anatomy of an LLM Request: Hidden Token Overheads

Developers often underestimate token consumption because API requests contain significant invisible metadata beyond the user's raw prompt:

┌─────────────────────────────────────────────────────────────┐
│ 1. System Prompt (Instructions, Guardrails, Formatting)    │ ~500 - 2,000 tokens
├─────────────────────────────────────────────────────────────┤
│ 2. Tool / Function Calling Definitions (JSON Schemas)       │ ~800 - 3,500 tokens
├─────────────────────────────────────────────────────────────┤
│ 3. Conversation History (Multi-Turn Messages & Tool Results)│ ~2,000 - 30,000 tokens
├─────────────────────────────────────────────────────────────┤
│ 4. Current User Query                                       │ ~50 - 500 tokens
├─────────────────────────────────────────────────────────────┤
│ 5. Reserved Output Buffer (max_tokens / reasoning_tokens)   │ ~1,000 - 8,192 tokens
└─────────────────────────────────────────────────────────────┘

• Tool Schema Overhead: Providing 5–10 detailed tool definitions (with JSON Schema properties and descriptions) consumes 1,000–3,000 input tokens on every single request, even if the model chooses not to invoke any tool.
• Reasoning Tokens (o1, o3, R1): Extended thinking models generate internal reasoning tokens that count toward both the output token limit and billable generation costs.

---

3. Programmatic Token Counting

Node.js / TypeScript (js-tiktoken)

import { encodingForModel, getEncoding } from 'js-tiktoken';

// 1. Get tokenizer for specific model
const enc = encodingForModel('gpt-4o'); // uses o200k_base

const prompt = 'Analyze this SQL query for indexing bottlenecks: SELECT * FROM orders;';
const tokens = enc.encode(prompt);

console.log(`Token count: ${tokens.length}`);
console.log('Token IDs:', tokens);

// Always free resources when complete
enc.free();

Python (tiktoken)

import tiktoken

def count_message_tokens(messages, model="gpt-4o"):
    """Accurately calculates token count including message delimiters."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        encoding = tiktoken.get_encoding("cl100k_base")
    
    num_tokens = 0
    for message in messages:
        num_tokens += 3  # every message follows <|start|>{role/name}\n{content}<|end|>\n
        for key, value in message.items():
            num_tokens += len(encoding.encode(str(value)))
    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
    return num_tokens

messages = [
    {"role": "system", "content": "You are a database optimization expert."},
    {"role": "user", "content": "How do B-Tree indexes handle UUIDv7 primary keys?"}
]

print(f"Total prompt tokens: {count_message_tokens(messages)}")

---

4. Context Budgeting & Cost Optimization Strategies

1. Leverage Prompt Caching (50% to 90% Savings)

Both OpenAI and Anthropic support automatic or explicit Prompt Caching. When you place static context (System Prompt, OpenAPI / MCP Tool definitions, large codebase documentation) at the very beginning of the prompt, subsequent requests that share the exact prefix bypass recomputation:

• Anthropic Prompt Caching: Cuts input token cost by 90% and reduces latency by up to 80% on cached prefixes.
• OpenAI Prompt Caching: Automatically applies a 50% discount on prompts over 1,024 tokens that share a prefix.

┌─────────────────────────────────────────────────────────────┐
│ [CACHED PREFIX - 90% DISCOUNT]                              │
│ • System Instructions                                       │
│ • Tool Schemas (JSON Schema)                                │
│ • Reference Knowledge Base                                  │
├─────────────────────────────────────────────────────────────┤
│ [DYNAMIC TAIL - FULL PRICE]                                 │
│ • Recent user turn & ephemeral context                      │
└─────────────────────────────────────────────────────────────┘

2. Context Window Sliding & Pruning

• Sliding Window: Keep only the last $N$ turns of chat history, summarizing older turns into a compact 100-token memory block.
• Tool Output Truncation: Never pass unparsed 500 KB API payloads directly into conversation turns. Filter responses to relevant keys before returning data to the agent.

Tip: Accurately calculate token counts across GPT-4o, Claude, and Llama vocabularies and forecast API generation expenses with the DevFlow AI Token Counter and AI Cost Calculator.