What Is Inference Speed in LLM? A Practitioner’s Guide
I got a call from a CTO in March 2026. His team had spent six months fine-tuning a 70B parameter model for their customer support pipeline. The accuracy was great. But when they put it into production, each response took 18 seconds. Users closed the chat before the AI finished typing.
That’s the gap between a demo and a product. That gap is inference speed.
Inference speed in large language models is the time it takes from when you send a prompt to when the last token is generated. Measured in milliseconds per token, or tokens per second (TPS). Simple concept. Brutally hard to optimize.
In this guide I’ll break down what influences that number, how to measure it, and what actually works when you’re shipping to real users. Not textbook theory — things I’ve learned the hard way building production AI systems at SIVARO since 2018.
Why Most People Get Inference Speed Wrong
Most engineers think inference speed is purely about hardware. Get a bigger GPU, go faster.
Reality is messier. I’ve seen a well-tuned model on an A10 run faster than a sloppy deployment on an H100. Inference speed is a system property — model architecture, memory bandwidth, quantization, batching, serving framework, even the shape of your input. Change one variable and your latency graph flips.
The other mistake: people conflate time to first token (TTFT) with total generation time. For interactive chat, TTFT matters most — users feel the delay before the first word appears. For batch processing, it’s all about throughput. Know which you’re optimizing for before you start.
The Anatomy of a Single Inference
Let’s walk through what happens when you ask a model a question. This is essential context for understanding speed.
- Tokenization – your prompt gets split into tokens (words or subwords). A 100-word prompt might become ~130 tokens.
- Prefill (or context encoding) – the model processes all input tokens in parallel. This is compute-bound. For long inputs, TTFT is dominated by this step.
- Autoregressive decoding – the model generates one token at a time, feeding the previous output back in. Each step is memory-bandwidth bound.
- Detokenization – back to human-readable text.
For a 256-token response, you do one prefill pass and 256 tiny forward passes. That asymmetry is why small changes in batching or KV-cache management have huge effects.
Here’s a quick code snippet to measure raw generation time with Hugging Face transformers (not production-ready, but fine for intuition):
python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import time
model_name = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
prompt = "Explain quantum computing in one paragraph."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
# Measure generation
t0 = time.time()
outputs = model.generate(**inputs, max_new_tokens=128)
t1 = time.time()
latency = t1 - t0
tokens_generated = outputs.shape[1] - inputs.input_ids.shape[1]
print(f"Generated {tokens_generated} tokens in {latency:.2f}s → {tokens_generated/latency:.1f} tokens/s")
Run that on an A100 and you’ll see ~30–50 tokens/s for a 7B model. On CPU? Maybe 0.5 tokens/s. That’s the hardware lever. But we’re just getting started.
The Five Levers That Control Speed
1. Model Size and Architecture
This is the obvious one. A 7B model is roughly 4x faster than a 30B model on the same hardware — if you keep everything else equal. But that’s rarely the right trade.
Larger models have better reasoning capability. The question is: do you need that capability, or can you get away with a smaller model plus a retrieval-augmented generation (RAG) pipeline? What Factors Influence LLM Inference Speed? lists parameter count as the #1 factor, but I’d argue context length is equally brutal. A 128K context window doubles the KV cache size by 8x compared to 16K. That memory pressure slows everything down.
Architecture matters too. MoE (mixture of experts) models like Mixtral 8x7B have sparse activation — only ~12B parameters active per forward pass, so inference speed is closer to a 12B dense model than a 56B one. That’s why MoE exploded in 2024–2025.
2. Quantization: Speed at a Cost
Quantization reduces the precision of model weights. FP16 → INT4 shrinks memory footprint by 4x. Smaller weights = more can fit in GPU SRAM = less memory bandwidth bottleneck = faster generation.
I ran a test in April 2026 with Llama 3 70B on a single A100-80GB. Unquantized (FP16): 15 tokens/s. AWQ 4-bit: 42 tokens/s. Nearly 3x improvement. The trade-off? Slight accuracy drop — about 0.5% on MMLU. For most production use cases, that’s fine. For medical or legal reasoning benchmarks, it might not be.
LLM Inference Optimization | Speed, Cost & Scalability covers quantization techniques in detail. The key insight: don’t quantize blindly. Test on your actual distribution.
3. Batching: The Hidden Lever
Serving multiple requests together amortizes the prefill cost and increases hardware utilization. Dynamic batching (grouping requests as they arrive) is table stakes. But the real trick is continuous batching — letting new requests join mid-generation.
vLLM popularized this in 2023. Before that, systems waited for a full batch to finish before accepting new work. Continuous batching can push throughput 2-4x higher on the same hardware. Real-World LLM Inference Benchmarks show vLLM beating naive Hugging Face pipelines by 5x on throughput for Llama 2 70B.
But batching increases latency for individual requests. If you need sub-200ms responses, you might run at batch size 1. Know your SLA.
4. Decoding Strategies
Greedy decoding (always pick the most probable token) is fastest. Sampling with temperature is slower because you need to compute probabilities and then sample. Beam search is a disaster for latency — multiple parallel sequences.
Most people don’t need beam search. For chat, simple top-k or top-p sampling works fine and is nearly as fast as greedy. I’ve seen teams throw beam search at summarization tasks where greedy + a slightly better model would have been faster and more accurate.
Also: speculative decoding. A small draft model generates multiple tokens; the large model verifies them in one forward pass. If the draft is correct (high acceptance rate), you get 2-3x speedup. Google’s Medusa and DeepMind’s self-speculative decoding are both production-ready as of 2025. We use a variant at SIVARO for a financial analysis pipeline.
5. Hardware Selection and Memory Bandwidth
GPUs differ not just in compute (FLOPS) but in memory bandwidth. An A100 has 2 TB/s bandwidth. An H100 has 3.35 TB/s. An RTX 4090 has 1 TB/s. For autoregressive decoding, every new token requires reading the entire model’s weights from memory into the compute units. Bandwidth is the bottleneck, not compute.
So an H100 doesn’t give you 1.67x the compute speedup — it gives you 1.67x the memory bandwidth speedup, which translates almost linearly to tokens/s for batch-size-1 inference. At large batch sizes, compute starts to matter. But most production workloads have small batches.
Groq’s LPU (language processing unit) architecture flips this — they use SRAM instead of HBM, achieving insane bandwidth. In 2025 they demoed 500 tokens/s on Llama 3 70B. That’s 10x faster than a top-tier GPU. But it’s a specialized chip; you can’t run arbitrary models.
Benchmarking: Measure What Matters
Don’t trust vendor benchmarks. They optimize for marketing numbers. LLM Inference Benchmarking - Measure What Matters makes a good point: you need to test on your own prompt lengths, output lengths, request patterns, and concurrency.
Standard benchmarks from dmatora/LLM-inference-speed-benchmarks give you a baseline. But they use fixed prompt/response lengths (e.g., 512/128). Your real traffic might be 2000/500. Test that.
Here’s a more realistic benchmark script (using the OpenAI-compatible API of a local vLLM server):
python
import time, concurrent.futures, requests, statistics
def make_request(prompt: str, max_tokens: int) -> float:
payload = {
"model": "llama3-70b",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens,
"temperature": 0.7
}
t0 = time.time()
resp = requests.post("http://localhost:8000/v1/chat/completions", json=payload)
t1 = time.time()
resp.raise_for_status()
return t1 - t0
# 10 concurrent requests, each generating 256 tokens
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
futures = [executor.submit(make_request, "Write a short story about a robot.", 256) for _ in range(10)]
latencies = [f.result() for f in futures]
print(f"P50 latency: {statistics.median(latencies)*1000:.0f} ms")
print(f"P99 latency: {sorted(latencies)[-2]*1000:.0f} ms")
Run that on different serving stacks. I’ve seen vLLM handle 10 concurrent requests on a single A100 at ~1.2s median latency while TGI (Text Generation Inference from Hugging Face) hit 2.1s. The gap widens at higher concurrency.
What to track:
- Time to first token (TTFT) – target < 500ms for interactive use.
- Inter-token latency (ITL) – time between tokens. For streaming, target < 30ms.
- Tokens per second (TPS) – either per-request or overall throughput.
- End-to-end latency – for synchronous requests, measure total time.
- Cost per token – dollars per million tokens. IBM’s LLM inference overview includes cost implications.
The Trade-Offs You Can’t Escape
Speed vs. Quality
Faster inference often means worse output. Quantization drops a few points on benchmarks. Smaller models are less capable. Greedy decoding produces repetitive text. There’s no free lunch.
But the cost of quality isn’t just accuracy — it’s user experience. A 10-second response that’s 1% more accurate is worse than a 2-second response that’s 1% less accurate, if the user leaves because it’s slow.
Speed vs. Cost
More GPUs = more speed. But also more money. Spot instances can cut costs 60-70%. For batch workloads, that’s fine. For real-time, you risk preemption. At SIVARO we run a mix: reserved instances for latency-critical paths, spot for offline summarization.
Speed vs. Memory
Longer contexts blow up the KV cache. At sequence length 128K, each request might consume 20+ GB of GPU memory just for the cache. That reduces your effective batch size. You can use sliding window attention (like Mistral) or sparse attention to trade recall for speed.
I watched a startup in 2024 try to serve a 70B model with 128K context on a single H100. They got 2 tokens/s. After switching to a sliding window of 4096, they hit 18 tokens/s — and their QA accuracy dropped 8%. They went back to the slow version for their legal product. Sometimes you can’t compress.
Production Patterns That Actually Work
Pattern 1: Route Short vs. Long Requests
Not all requests need the same model. Short queries (e.g., “What’s the weather?”) can go to a 1B Phi-3 variant. Complex reasoning goes to the 70B. Use a classifier or a small router model to decide. We’ve seen average latency drop 70% while keeping quality on hard questions.
Pattern 2: Streaming with Progressive Output
Send the first token as fast as possible (even if it’s “…”). Then stream the rest. Users perceive speed based on TTFT, not total time. A 200ms TTFT with a slow 20ms ITL feels faster than a 1s TTFT with a fast 5ms ITL. We’ve A/B tested this.
Pattern 3: Use a Inference Server, Not a Framework Wrapper
Writing your own FastAPI endpoint around a raw model is fine for demos. For production, use vLLM, TensorRT-LLM, or TGI. They handle memory management, batching, and KV-cache paging. Rolling your own is a path to pain.
What’s Changed Recently (2025-2026)
The biggest shift? Inference as a service commoditization. Groq, Together AI, Fireworks, and others offer API endpoints with optimization layers you’d never build yourself. For many teams, it’s cheaper to pay per token than to manage H100 clusters.
That doesn’t mean you ignore speed — it means you can buy speed instead of building it. But you still need to understand the numbers to choose a provider. Real-World LLM Inference Benchmarks showed Fireworks beating Groq on throughput for certain models in early 2026 due to better continuous batching. The landscape shifts monthly.
Also: AMD’s MI300X has become a viable alternative to NVIDIA for inference. In our tests in April 2026, MI300X matched H100 on throughput for large batch sizes (128+). For small batches, H100 still won by 30%. But the price difference is significant.
FAQ
Q: What is inference speed in LLM?
It’s the rate at which a model generates tokens from a prompt. Measured in tokens per second (TPS) or latency per token. Affected by model size, hardware, quantization, batching, and decoding strategy.
Q: What’s a good inference speed for production?
For interactive chat, aim for at least 20 TPS with TTFT under 500ms. For batch processing, 50+ TPS per GPU is reasonable. Sub-10 TPS is painful for users.
Q: Does increasing batch size always improve throughput?
Up to a point. After GPU memory fills, increasing batch size causes out-of-memory errors or slower prefill. Find the sweet spot by benchmarking.
Q: Should I use INT4 quantization?
If you care about speed and your quality loss is acceptable, yes. Always evaluate on your tasks first — some models degrade more than others.
Q: Why is my TTFT so high on long inputs?
The prefill phase processes all input tokens in parallel. On an H100, prefill for 4000 tokens takes ~100ms. On an A10, it might take 400ms. Shorten your input (e.g., via RAG) or upgrade hardware.
Q: Is speculative decoding ready for production?
Yes. Multiple frameworks (vLLM, TensorRT-LLM) support it. You need a good draft model. Acceptance rates of 70-80% are common, yielding 2x speedups.
Q: How does Flash Attention help?
Flash Attention reduces the quadratic memory complexity of attention to linear. It allows longer contexts without slowing down from memory paging. Most modern frameworks use it by default.
Q: Can I run a 70B model on a single GPU?
With quantization (4-bit) and limited batch size, yes. Expect 15-30 tokens/s on an H100. On an A100, 10-15 tokens/s.
The Bottom Line
You can’t fix slow inference by throwing money at GPUs. Not anymore. The difference between a fast and slow system is architecture, not just acceleration.
Understand your latency budget. Benchmark on your data. Use quantization, batching, and speculative decoding in that order. And when you can, buy inference as a service — the optimization work is already done.
The CTO I talked to in March? We rebuilt their pipeline with continuous batching, 4-bit quantization, and a smaller draft model. Their 18-second responses dropped to 1.2 seconds. Users stopped closing the chat.
That’s what inference speed is. Not a number on a benchmark. A business lever.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.