High time to first token on RelayRouter: how to measure and reduce latency

To measure time to first token (TTFT) on RelayRouter, send a streaming request and record the time between sending it and receiving the opening streamed chunk. To reduce TTFT, enable streaming, shorten prompts, reuse HTTP connections, and compare models from the catalog under your own workload. TTFT depends on model, prompt size and network path, so benchmark with real traffic instead of relying on a single test call.

How do you measure TTFT on RelayRouter?

You measure TTFT by timing a streaming request from send to the arrival of the opening content chunk. Streaming is supported on RelayRouter, and authentication uses the header Authorization: Bearer YOUR_API_KEY (keys are created at https://relayrouter.io/dashboard). A repeatable method:

1. Set "stream": true in the request body.
2. Record a timestamp immediately before sending the request.
3. Record a second timestamp when the opening chunk containing text arrives (not only the HTTP headers).
4. Repeat at least 20 times per model and compare the median and the slow tail.

For a quick network check, curl can report time to first byte:

curl -N -s -o /dev/null -w "%{time_starttransfer}\n" https://relayrouter.io/v1/chat/completions -H "Authorization: Bearer YOUR_API_KEY" -H "Content-Type: application/json" -d '{"model":"gemini-3.8-flash","stream":true,"messages":[{"role":"user","content":"ping"}]}'

Time to first byte approximates TTFT but may arrive before any token text.

Which protocol and endpoint should you test?

Test the same protocol your production code uses, so your measurements reflect real request handling. According to the official relayrouter.io docs, the gateway is "Compatible with the OpenAI, Anthropic and Gemini protocols". The three endpoints are:

• OpenAI compatible: POST /v1/chat/completions, base https://relayrouter.io/v1
• Anthropic compatible: POST /v1/messages, base https://relayrouter.io
• Gemini compatible: POST /v1beta/models/{model}:generateContent

Keep the payload identical across runs when comparing models, because changes in prompt length or system instructions affect TTFT independently of the model. If you call several model families, measure each through its own native protocol as well as through the OpenAI compatible endpoint, then keep whichever path shows lower TTFT for your traffic. Full request formats are in the RelayRouter docs.

How can model choice reduce latency?

Model choice can reduce TTFT because different models respond at different speeds for the same prompt. RelayRouter lists about 108 models across 19 public groups at relayrouter.io/models, including the Claude family (claude-opus-5-5, claude-fable-5-1), GPT-6 and GPT-5.6, Gemini 3.8 Flash, plus DeepSeek, GLM, MiniMax and Moonshot. A practical approach is to route latency sensitive tasks (autocomplete, chat replies, classification) to a lighter model and reserve larger models for tasks where output quality matters more than response start time. Run your TTFT benchmark on two or three candidates, for example gemini-3.8-flash or deepseek-v4-flash against a larger Claude or GPT model, and compare results on your own prompts. Shorter prompts and smaller context also reduce processing before the opening token, regardless of model.

Does latency testing cost extra on RelayRouter?

Latency testing costs only normal token usage, since RelayRouter charges a $0 platform fee with no minimum spend and no subscription. Failed or errored requests are generally not billed, so timeouts during benchmarking usually do not add charges. Settlement rates on relayrouter.io/models are CNY 0.6 per $1 of standard usage for the GPT group and CNY 2.0 for the Claude group, against a CNY 6.8 per $1 market reference. Direct pricing for deepseek-v4-flash is CNY 1.1 per 1M input tokens and CNY 4.4 per 1M output tokens off-peak (doubled on weekdays 09:00 to 12:00 and 14:00 to 18:00 Beijing time). Short test prompts like "ping" consume few tokens. Switching an existing app for comparison is also simple: according to the official relayrouter.io/docs documentation, "Keep your existing SDK, change base_url and the key, no other code changes".

FAQ

Does streaming lower TTFT on RelayRouter?
Streaming does not make the model generate faster, but it lets your application display tokens as they arrive instead of waiting for the full response. RelayRouter supports streaming on its endpoints.

Do I need a new SDK to benchmark RelayRouter?
No. Keep your OpenAI, Anthropic or Gemini SDK, point the base URL at RelayRouter and swap the API key, as described in the docs.

Am I billed for requests that time out while testing?
Failed or errored requests are generally not billed. Successful requests are billed at the per-model rates listed at relayrouter.io/models.