Measuring LLM latency honestly: TTFT, total time, and why tokens/s needs at least 50 tokens
Published · 6 min read
Disclosure: this article is by the team behind Ezkielyna, an independent AI API gateway. The numbers here come from its public status feed, which anyone can read without a key. Drafted with AI assistance and edited by the team.
"Model X is fast" is usually a single request, timed once, with a two-word prompt. That number tells you very little about your app.
We publish per-model latency and uptime from real gateway traffic at api.ezkielyna.store/status. The raw JSON is at /status/models.json. Building it forced us to write down rules for what counts and what doesn't. Here are those rules, why each one exists, and what the data looked like on 12 Nov 2026.
Four numbers, four different questions
| Metric | Question it answers | How we compute it |
|---|---|---|
| TTFT (time to first token) | "How long until the user sees something?" | Streamed answers only, median (p50) and 90th percentile (p90), last 24 h |
| Total latency | "How long until the answer is complete?" | Median over the last 24 h. The gateway logs total time in whole seconds |
| TPS (output tokens per second) | "Once it starts, how fast does it write?" | Streamed answers with at least 50 output tokens and at least 2 s of generation, median, last 24 h |
| Success / uptime | "Will my request work at all?" | % of requests that did not fail with a 5xx on the gateway's or the upstream's side, over 24 h, 7 d and 30 d |
These four do not collapse into one number. A model with a short TTFT but slow TPS feels fast in a chat box and slow in a long code edit. A model with a high TPS and a long TTFT is the reverse.
Rule 1: TTFT only counts if you streamed
Without streaming, the first byte you receive is the whole answer, so "TTFT" is really total latency. Mixing the two makes a model look slower than it is. We compute TTFT only from streamed answers.
Rule 2: tokens per second needs enough tokens
Short answers produce garbage TPS. Say a model spends 3 s before the first token and then writes 8 tokens. Divide 8 tokens by the total time and you get a speed under 3 tokens/s, but that mostly measures waiting, not writing.
Our TPS rule therefore keeps only answers with at least 50 output tokens and at least 2 s of generation. Some models never qualify. In the 12 Nov snapshot, minimax-m3 answered 188 requests in 24 h with a 100% success rate and a p50 TTFT of 2,612 ms. Its TPS field is empty, because too few of those answers were long enough to measure.
An empty field is more honest than a made-up one.
Rule 3: report p50 and p90, never the mean
Latency distributions have long tails. Here is our highest-traffic model on 12 Nov 2026:
| Model | Requests (24 h) | TTFT p50 | TTFT p90 | Total p50 | TPS p50 |
|---|---|---|---|---|---|
| deepseek-v4.1-flash | 7,340 | 3,844 ms | 10,883 ms | 5.0 s | 119.5 |
| deepseek-v4.1-flash-1017 | 9,065 | 4,029 ms | 13,092 ms | 5.0 s | 125.4 |
| deepseek-v4-pro | 677 | 4,568 ms | 6,667 ms | 6.0 s | 83.5 |
| glm-5.3 | 208 | 3,482 ms | 7,943 ms | 4.0 s | 62.4 |
| gpt-6.1-terra | 386 | 9,810 ms | 25,289 ms | 13.0 s | 49.2 |
| claude-opus-5.5 | 107 | 4,574 ms | 8,996 ms | 9.0 s | 143.9 |
For DeepSeek V4.1 Flash, p90 TTFT is about 2.8 times the p50. A mean would sit somewhere in between and describe nobody's real experience. If your product has a timeout, the p90 is the number to design for.
Rule 4: say what your prompts look like
Why is TTFT around 4 seconds for a "Flash" model? Look at the traffic, not the model. Over the 7 days to 12 Nov 2026, requests to deepseek-v4.1-flash averaged about 112,750 prompt tokens and about 719 completion tokens each: 4,487,326,832 prompt tokens over 39,799 requests. That is coding-agent traffic, where a whole repository of context goes in and a short edit comes out.
Prefill on 100k+ tokens takes time. A benchmark that sends "hi" will show a much smaller TTFT and tell you nothing about agent workloads. Two caveats:
- This is a mean (a median isn't available in the public file).
- The prompt count includes tokens that may have been served from cache.
When you read anyone's latency chart, including ours, ask what prompt sizes it came from.
Rule 5: only count failures that are the service's fault
Our success rate counts a request as failed only for a 5xx on the gateway's or the upstream's side. A 401 for a wrong key or a 400 for a malformed request is the caller's problem, not the model's. Counting those would punish popular models for their users' typos.
The flip side: 5xx failures are counted even when they are embarrassing. In the same snapshot, minimax-m3.1-flash had a 24 h success rate of 39.29% over 56 requests, and 81.91% over 7 days. That is a bad day, and the feed shows it. A status page that only ever shows green is a marketing page.
Rule 6: the window changes the story
Pick the window that matches your question:
gpt-5.6-lunashowed 99.66% uptime over 7 days but 81.73% over 30 days on 12 Nov 2026. Both are true; the 7-day figure describes the route today, the 30-day figure describes its history.- Low-traffic models are flagged
low_datawhen they have fewer than 20 answers in 24 h, and we hide their latency rather than print a number from three requests. - Of 279 listed models (after we removed some ids from public listings), 95 had enough traffic for latency figures in this snapshot.
Rule 7: measured where?
All of this is measured at the gateway, between the gateway and its upstreams, plus the gateway's own processing. Your users add their own network path on top. A gateway number is a lower bound for your app, not a promise.
Measure your own route in 40 lines
This uses the openai Python package against any OpenAI-compatible base URL, and applies the same rules: stream, time the first token, and only report TPS when there are enough tokens.
import os, sys, time, statistics
from openai import OpenAI
client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])
MODEL = sys.argv[1]
PROMPT = open(sys.argv[2]).read() # use a prompt the size of your real ones
RUNS = int(os.environ.get("RUNS", "10"))
ttft, total, tps = [], [], []
for _ in range(RUNS):
t0 = time.perf_counter()
first = None
out_tokens = 0
stream = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": PROMPT}],
stream=True,
stream_options={"include_usage": True},
max_tokens=400,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content and first is None:
first = time.perf_counter()
if chunk.usage: # the usage chunk has empty choices
out_tokens = chunk.usage.completion_tokens
end = time.perf_counter()
if first is None:
continue # no content at all: count it as a failure, not a fast answer
ttft.append((first - t0) * 1000)
total.append(end - t0)
gen = end - first
if out_tokens >= 50 and gen >= 2:
tps.append(out_tokens / gen)
def p(xs, q):
return statistics.quantiles(xs, n=10)[q - 1] if len(xs) >= 10 else None
print(f"runs ok: {len(ttft)}/{RUNS}")
print(f"TTFT ms p50={statistics.median(ttft):.0f} p90={p(ttft, 9)}")
print(f"total s p50={statistics.median(total):.2f}")
print(f"TPS p50={statistics.median(tps):.1f}" if tps else "TPS: not enough long answers")Run it 10+ times, at the hour your users are active, with a prompt from your real traffic. It costs real tokens, so keep RUNS and max_tokens small.
What we'd ask of any provider's latency page
- Is TTFT from streamed requests?
- Is TPS filtered for short answers?
- Do they show percentiles, not means?
- Which failures count?
- What are the window and the sample size?
- Where was it measured from?
- Are bad days visible?
If a page can't answer these, treat its numbers as marketing.
Our answers are in the notes block at the top of /status/models.json. Corrections are welcome.
Sources
- Ezkielyna status feed, snapshot 2026-11-12 01:30 WIB: https://api.ezkielyna.store/status/models.json
- Ezkielyna weekly leaderboard, 5-12 Nov 2026: https://api.ezkielyna.store/status/leaderboard.json
- OpenAI streaming usage chunk (
include_usage): https://community.openai.com/t/usage-stats-now-available-when-using-streaming-with-the-chat-completions-api-or-completions-api/738156
Ezkielyna is an independent API gateway: one OpenAI- and Anthropic-compatible key for many AI models, prepaid with QRIS, USDT or Binance Pay, no credit card needed. It is not affiliated with the companies that make the models.