Cost Optimization Offline Testing

Cost levers ranked by leverage: prompt cache, model right-sizing, context trimming, batch APIs, retrieval-before-LLM. Each lever has an offline harness pattern so you can measure the win before shipping.

Interview talking points

• Order of cost levers. 1) Prompt cache, 2) model right-sizing via Arena, 3) trim context, 4) batch what you can, 5) retrieval-before-LLM, 6) self-host (last resort).
• Why offline first? Production A/B is the slowest measurement loop; offline = same query set + same prompt = clean delta.
• Cache-hit math. Anthropic prompt cache: cached tokens ~10% of normal price, ~5x faster TTFT. Justifies a 1024+ token system prompt if you reuse it.
• Model arena methodology. Blind voting, locked promotion rule (cheapest model that wins ≥X% of votes ships).

Files in this folder

Back to the home page.