How to Reduce LLM Latency

A proper technical breakdown of where LLM latency actually comes from, with experiments instead of hand-waving. The core lesson: prefill (processing your prompt) is parallel and compute-bound, while decode (generating tokens) is sequential and memory-bound — and decode is almost always your bottleneck. If you're building agentic products, the cost-multiplication section alone justifies the time.

• Generating one output token costs roughly 300x more time than processing one input token — cap max_tokens before you spend weeks optimizing your retrieval pipeline.
• In a 5-step agent loop, cost multiplied 11.9x because each step resends the full prior history; context management is the difference between a viable product and one killed by its bill.
• Small requests waste GPU: 143 tokens/sec at 25 tokens vs 2,271 tokens/sec at 1,205 tokens — continuous batching exists to fix exactly this.
• GPU warm-up is routinely skipped in production and cold starts quietly destroy SLAs.

Part of Issue Nº 001: LLMs for judgment, code for counting: scaling to 500K sensors on 300x fewer tokens