"OpenAI-compatible" is a spectrum: 10 quirks we hit running a multi-upstream gateway
Published · 7 min read
Disclosure: this article is by the team behind Ezkielyna, an independent AI API gateway that speaks the OpenAI, Anthropic and Responses formats in front of many upstreams. Every quirk below is one we hit in production. No upstream is named, because the point is the pattern, not the vendor. Drafted with AI assistance and edited by the team.
"OpenAI-compatible" usually means "the happy path works". You send one user message with no tools and no streaming, and you get text back. Real clients do much more than that: coding agents, n8n flows, chat UIs that replay history. Each of them leans on a different corner of the spec, and each upstream fills those corners in its own way.
Here are ten corners, with the fix we settled on for each one.
1. tool_calls: null in replayed history
Many clients replay the whole conversation, and some serialise assistant turns as {"role":"assistant","content":"…","tool_calls":null}. Some upstreams accept that. Others reject the whole request with a 400 because null is not an array, and a few also reject tool_calls: [].
Fix: before forwarding, drop tool_calls when it is null or an empty array. Do the same for empty content arrays on tool-only turns.
2. Tool arguments: a string, not an object, and index matters
In the OpenAI format, function.arguments is a JSON string. Some upstreams return a parsed object instead, and strict SDKs then fail to parse it.
While streaming, tool calls arrive as deltas, and each delta needs an index so the client can stitch fragments together. When index is missing, parallel tool calls merge into one broken call.
Fix:
- Normalise
argumentsto a string. - Fill in
indexwhen the upstream omits it. - Add a test with two parallel tool calls, because one call alone hides this bug.
3. Three (or four) ways to ask for "think harder"
The same intent travels in different shapes:
reasoning_effort: "high"in Chat Completions.reasoning: { effort: "high" }in the Responses API and some routers.thinking: { type: "enabled", budget_tokens: N }in Anthropic Messages.- Some providers now take two of these together. DeepSeek's own reasoning guide shows
thinking: {type: "enabled"}next toreasoning_effortin one request.
The nasty part is a silent failure. We saw one upstream return an empty answer that was still billed when reasoning_effort arrived together with stream: true, with no error, just nothing.
Fix:
- Translate the knob per upstream rather than passing it through blindly.
- Treat "200 OK, zero output tokens, non-zero input tokens" as a failure in your own monitoring.
4. Reasoning text coming back in, or not
Reasoning models stream their thinking in a separate field. The field is often reasoning_content, and some providers use reasoning. What you send back on the next turn varies:
- Some upstreams reject a request that includes the previous turn's reasoning text.
- Others expect it back during multi-step tool calling.
- Clients built against one behaviour break on the other.
Fix:
- Make it a per-model setting in the gateway, not a global one.
- Test a three-turn tool conversation on every reasoning model you add, not just a single turn.
5. The usage chunk: empty choices, or not
With stream_options: {"include_usage": true}, OpenAI sends one extra chunk before data: [DONE]. That chunk carries the totals, and its choices array is empty. Not everyone matches that:
- LiteLLM issue #28735 reports a proxy emitting the usage chunk with a non-empty
choices. LangChain.js then double-counted tokens. - llama.cpp issue #15443 reports usage arriving on the final content chunk instead of a separate one.
- Some upstreams never send usage while streaming.
Fix:
- Clients: never assume
choices[0]exists on every chunk, and readusagefrom whichever chunk carries it. - Gateways: if the upstream sends no usage, count the tokens yourself and emit one OpenAI-shaped usage chunk. Billing must never depend on the upstream remembering.
6. How a stream ends
Each format ends differently:
- Chat Completions:
data: [DONE]. - Anthropic Messages: named events ending with
message_stop. Anthropic's versioning page notes that the 2023-06-01 version "removed unnecessarydata: [DONE]event". - Responses API: typed events such as
response.completed.
Upstreams sometimes omit the terminator, send it twice, or close the socket without one after a long pause.
Fix:
- In the gateway: emit exactly one terminator in the format the client asked for, whatever the upstream did.
- In the client: treat a closed connection without a terminator as an error, not as success.
7. Cached tokens: same idea, different fields, different maths
| Format | Field | Included in input count? |
|---|---|---|
| OpenAI | usage.prompt_tokens_details.cached_tokens | yes, part of prompt_tokens |
| Anthropic | cache_read_input_tokens, cache_creation_input_tokens | no, reported separately from input_tokens |
| DeepSeek | prompt_cache_hit_tokens, prompt_cache_miss_tokens | hit + miss = input |
If you translate Anthropic usage into OpenAI usage and forget to add the cache fields into prompt_tokens, cost dashboards under-report. If you add them twice, they over-report. We have seen both mistakes in the wild.
Fix: convert to one internal shape (input_uncached, input_cache_read, input_cache_write, output) and derive each client format from it.
8. max_tokens vs max_completion_tokens
OpenAI's reasoning models want max_completion_tokens. An OpenAI staff reply on their forum confirms the o1 series only supports that name, while max_tokens remains for older models. Many OpenAI-compatible upstreams still only understand max_tokens.
Anthropic's newest API version (2026-10-21) adds a new trap: it removes temperature, top_p and top_k. "A request that sends any of them returns a 400 error on every model."
Fix:
- Accept both token-limit names and map them to whatever each upstream expects.
- Strip sampling parameters for upstreams that reject them, instead of passing a 400 back to a user who never chose those settings.
9. Body limits and required headers
Two failures look like random 400s:
- Body size. A chat with a few base64 screenshots can exceed an upstream's request-size limit. Some return 413; others return a generic 400 with no hint.
- Headers. Anthropic requires an
anthropic-versionheader on every request. Claude Code's gateway compatibility guide asks gateways to forwardanthropic-beta"verbatim; don't allowlist individual values, because the set changes with Claude Code releases."
Fix:
- Measure the body size before forwarding and return a clear 413 that names the limit.
- Pass
anthropic-*headers through unchanged, as Anthropic's guide says.
10. Three endpoints for "the same model", and the model field
The same model id may be reachable through:
/v1/chat/completions;/v1/responses, which Codex CLI's docs configure withwire_api = "responses";/v1/messages, which Claude Code uses.
In our public pricing snapshot of 12 Nov 2026, 284 of 313 models list the OpenAI chat endpoint type. Only 57 list the Anthropic or Responses types. "Works in my curl test" does not mean "works in my coding agent".
Also look at the model field in responses. A UC Berkeley paper presented at NeurIPS 2025 ("Are You Getting What You Pay For?") shows that silent substitution of a cheaper model is hard to detect from outside. Software checks alone are not reliable.
Fix:
- Publish which endpoint types each model supports.
- Return errors unmodified where you can. Claude Code's guide explains that its retry logic "matches on the upstream's error wording", so wrapping errors breaks recovery.
- State your routing rule in writing, for example "same model only": a fallback can change the route, never the model.
A checklist you can steal
Before you call an endpoint "compatible", test:
- [ ] history replay containing
tool_calls: null; - [ ] two parallel tool calls while streaming;
- [ ] each reasoning knob, streamed, checking for empty billed answers;
- [ ] a three-turn tool conversation on reasoning models;
- [ ]
include_usagewith a parser that tolerates emptychoices; - [ ] stream termination, and a connection drop mid-stream;
- [ ] cache fields: send the same long prefix twice and compare the counts;
- [ ] both token-limit names, plus sampling parameters on a strict upstream;
- [ ] an oversized body, and missing or extra headers;
- [ ] each endpoint type you claim, and the
modelfield in the reply.
These checks are now an open-source CLI, llm-gateway-doctor: run npx -y llm-gateway-doctor --help and point it at any base URL with your own key. Our public per-model status feed shows which models are healthy right now, with no key needed: api.ezkielyna.store/status.
What did we miss? Open an issue on llm-gateway-doctor with the quirk that cost you a weekend.
Sources (checked 12 Nov 2026)
- DeepSeek reasoning guide: https://api-docs.deepseek.com/guides/reasoning_model
- DeepSeek context caching: https://api-docs.deepseek.com/guides/kv_cache
- OpenAI forum, usage in streaming: https://community.openai.com/t/usage-stats-now-available-when-using-streaming-with-the-chat-completions-api-or-completions-api/738156
- LiteLLM issue #28735: https://github.com/BerriAI/litellm/issues/28735
- llama.cpp issue #15443: https://github.com/ggml-org/llama.cpp/issues/15443
- Anthropic API versions: https://platform.claude.com/docs/en/api/versioning
- OpenAI forum, max_tokens vs max_completion_tokens: https://community.openai.com/t/why-was-max-tokens-changed-to-max-completion-tokens/938077
- Claude Code gateway compatibility guide: https://code.claude.com/docs/en/llm-gateway-protocol
- Codex advanced config: https://learn.chatgpt.com/docs/config-file/config-advanced
- Cai et al., NeurIPS 2025: https://arxiv.org/abs/2504.04715
- Ezkielyna pricing snapshot: https://api.ezkielyna.store/api/pricing (saved 12 Nov 2026 01:33 WIB)
Ezkielyna is an independent API gateway: one OpenAI- and Anthropic-compatible key for many AI models, prepaid with QRIS, USDT or Binance Pay, no credit card needed. It is not affiliated with the companies that make the models.