Measuring LLM latency honestly: TTFT, total time, and why tokens/s needs at least 50 tokens

Published · 6 min read

Disclosure: this article is by the team behind Ezkielyna, an independent AI API gateway. The numbers here come from its public status feed, which anyone can read without a key. Drafted with AI assistance and edited by the team.

"Model X is fast" is usually a single request, timed once, with a two-word prompt. That number tells you very little about your app.

We publish per-model latency and uptime from real gateway traffic at api.ezkielyna.store/status. The raw JSON is at /status/models.json. Building it forced us to write down rules for what counts and what doesn't. Here are those rules, why each one exists, and what the data looked like on 12 Nov 2026.

Four numbers, four different questions

MetricQuestion it answersHow we compute it
TTFT (time to first token)"How long until the user sees something?"Streamed answers only, median (p50) and 90th percentile (p90), last 24 h
Total latency"How long until the answer is complete?"Median over the last 24 h. The gateway logs total time in whole seconds
TPS (output tokens per second)"Once it starts, how fast does it write?"Streamed answers with at least 50 output tokens and at least 2 s of generation, median, last 24 h
Success / uptime"Will my request work at all?"% of requests that did not fail with a 5xx on the gateway's or the upstream's side, over 24 h, 7 d and 30 d

These four do not collapse into one number. A model with a short TTFT but slow TPS feels fast in a chat box and slow in a long code edit. A model with a high TPS and a long TTFT is the reverse.

Rule 1: TTFT only counts if you streamed

Without streaming, the first byte you receive is the whole answer, so "TTFT" is really total latency. Mixing the two makes a model look slower than it is. We compute TTFT only from streamed answers.

Rule 2: tokens per second needs enough tokens

Short answers produce garbage TPS. Say a model spends 3 s before the first token and then writes 8 tokens. Divide 8 tokens by the total time and you get a speed under 3 tokens/s, but that mostly measures waiting, not writing.

Our TPS rule therefore keeps only answers with at least 50 output tokens and at least 2 s of generation. Some models never qualify. In the 12 Nov snapshot, minimax-m3 answered 188 requests in 24 h with a 100% success rate and a p50 TTFT of 2,612 ms. Its TPS field is empty, because too few of those answers were long enough to measure.

An empty field is more honest than a made-up one.

Rule 3: report p50 and p90, never the mean

Latency distributions have long tails. Here is our highest-traffic model on 12 Nov 2026:

ModelRequests (24 h)TTFT p50TTFT p90Total p50TPS p50
deepseek-v4.1-flash7,3403,844 ms10,883 ms5.0 s119.5
deepseek-v4.1-flash-10179,0654,029 ms13,092 ms5.0 s125.4
deepseek-v4-pro6774,568 ms6,667 ms6.0 s83.5
glm-5.32083,482 ms7,943 ms4.0 s62.4
gpt-6.1-terra3869,810 ms25,289 ms13.0 s49.2
claude-opus-5.51074,574 ms8,996 ms9.0 s143.9

For DeepSeek V4.1 Flash, p90 TTFT is about 2.8 times the p50. A mean would sit somewhere in between and describe nobody's real experience. If your product has a timeout, the p90 is the number to design for.

Rule 4: say what your prompts look like

Why is TTFT around 4 seconds for a "Flash" model? Look at the traffic, not the model. Over the 7 days to 12 Nov 2026, requests to deepseek-v4.1-flash averaged about 112,750 prompt tokens and about 719 completion tokens each: 4,487,326,832 prompt tokens over 39,799 requests. That is coding-agent traffic, where a whole repository of context goes in and a short edit comes out.

Prefill on 100k+ tokens takes time. A benchmark that sends "hi" will show a much smaller TTFT and tell you nothing about agent workloads. Two caveats:

  • This is a mean (a median isn't available in the public file).
  • The prompt count includes tokens that may have been served from cache.

When you read anyone's latency chart, including ours, ask what prompt sizes it came from.

Rule 5: only count failures that are the service's fault

Our success rate counts a request as failed only for a 5xx on the gateway's or the upstream's side. A 401 for a wrong key or a 400 for a malformed request is the caller's problem, not the model's. Counting those would punish popular models for their users' typos.

The flip side: 5xx failures are counted even when they are embarrassing. In the same snapshot, minimax-m3.1-flash had a 24 h success rate of 39.29% over 56 requests, and 81.91% over 7 days. That is a bad day, and the feed shows it. A status page that only ever shows green is a marketing page.

Rule 6: the window changes the story

Pick the window that matches your question:

  • gpt-5.6-luna showed 99.66% uptime over 7 days but 81.73% over 30 days on 12 Nov 2026. Both are true; the 7-day figure describes the route today, the 30-day figure describes its history.
  • Low-traffic models are flagged low_data when they have fewer than 20 answers in 24 h, and we hide their latency rather than print a number from three requests.
  • Of 279 listed models (after we removed some ids from public listings), 95 had enough traffic for latency figures in this snapshot.

Rule 7: measured where?

All of this is measured at the gateway, between the gateway and its upstreams, plus the gateway's own processing. Your users add their own network path on top. A gateway number is a lower bound for your app, not a promise.

Measure your own route in 40 lines

This uses the openai Python package against any OpenAI-compatible base URL, and applies the same rules: stream, time the first token, and only report TPS when there are enough tokens.

python
import os, sys, time, statistics
from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])
MODEL = sys.argv[1]
PROMPT = open(sys.argv[2]).read()  # use a prompt the size of your real ones
RUNS = int(os.environ.get("RUNS", "10"))

ttft, total, tps = [], [], []
for _ in range(RUNS):
    t0 = time.perf_counter()
    first = None
    out_tokens = 0
    stream = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": PROMPT}],
        stream=True,
        stream_options={"include_usage": True},
        max_tokens=400,
    )
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content and first is None:
            first = time.perf_counter()
        if chunk.usage:  # the usage chunk has empty choices
            out_tokens = chunk.usage.completion_tokens
    end = time.perf_counter()
    if first is None:
        continue  # no content at all: count it as a failure, not a fast answer
    ttft.append((first - t0) * 1000)
    total.append(end - t0)
    gen = end - first
    if out_tokens >= 50 and gen >= 2:
        tps.append(out_tokens / gen)

def p(xs, q):
    return statistics.quantiles(xs, n=10)[q - 1] if len(xs) >= 10 else None

print(f"runs ok: {len(ttft)}/{RUNS}")
print(f"TTFT ms p50={statistics.median(ttft):.0f} p90={p(ttft, 9)}")
print(f"total s p50={statistics.median(total):.2f}")
print(f"TPS p50={statistics.median(tps):.1f}" if tps else "TPS: not enough long answers")

Run it 10+ times, at the hour your users are active, with a prompt from your real traffic. It costs real tokens, so keep RUNS and max_tokens small.

What we'd ask of any provider's latency page

  • Is TTFT from streamed requests?
  • Is TPS filtered for short answers?
  • Do they show percentiles, not means?
  • Which failures count?
  • What are the window and the sample size?
  • Where was it measured from?
  • Are bad days visible?

If a page can't answer these, treat its numbers as marketing.

Our answers are in the notes block at the top of /status/models.json. Corrections are welcome.


Sources

Ezkielyna is an independent API gateway: one OpenAI- and Anthropic-compatible key for many AI models, prepaid with QRIS, USDT or Binance Pay, no credit card needed. It is not affiliated with the companies that make the models.

About Ezkielyna API →