What is the best way to compare LLM API costs?
Calculate cost per task
Start with the provider's per-token prices for input and output. Estimate the average tokens per request for your use case. Multiply and add to get cost per request. Then adjust for retries: if a model fails 20% of the time and you retry, multiply by 1.2.
Also check for hidden costs: some providers charge for fine-tuning, embeddings, or data storage. Free tiers may have rate limits that force you to upgrade. Read the pricing page carefully.
- Input tokens × input price + output tokens × output price = base cost
- Add retry multiplier (e.g., 1.1–1.5x)
- Include any platform or minimum fees
- Compare cost per successful task, not per token
Run a side-by-side test
Take 50–100 real examples from your application. Run them through each candidate model. Measure accuracy (or quality score), latency, and total tokens used. Then calculate cost per correct answer.
This gives you a realistic comparison. A model that costs twice as much per token but needs half the retries and produces better answers may be cheaper overall. Document your results and revisit quarterly as prices change.
Common mistakes
- Comparing only the headline per-token price and ignoring output token costs.
- Forgetting to account for retries or failures, which increase effective cost.
- Not testing with your own data, leading to misleading conclusions from generic benchmarks.
