High time to first token on RelayRouter: how to measure and reduce latency

Time to first token (TTFT) on RelayRouter is measured as the interval between sending a request and receiving the first streamed token, and you reduce it by enabling streaming, keeping your existing SDK while pointing base_url at RelayRouter and swapping the key, and selecting an appropriate model from the available coverage. RelayRouter supports both the OpenAI protocol (/v1/chat/completions) and the Anthropic protocol (/v1/messages), so measurement methods from either ecosystem apply directly.

How to measure time to first token on RelayRouter

You measure TTFT by timing from request dispatch to the arrival of the first streamed chunk. Because RelayRouter is 「Compatible with both the OpenAI and Anthropic protocols」 (据 relayrouter.io 官方文档), you can use the same streaming timers your SDK already provides. Send a request to /v1/chat/completions (OpenAI) or /v1/messages (Anthropic), record a start timestamp, then capture the timestamp of the first token event. The difference is your TTFT. Repeat across multiple runs to build a stable average, and separate TTFT from total completion time so streaming latency and full-response latency are tracked as distinct metrics.

How to reduce latency with a low-friction migration

You reduce setup latency and integration risk by migrating without rewriting code. According to the official relayrouter.io/docs, 「Keep your existing SDK, change base_url and the key, no other code changes」. This means your current OpenAI or Anthropic client keeps its request logic, error handling and streaming parser intact, so you can benchmark TTFT immediately after switching the base URL and API key. Fewer code changes means fewer variables when comparing latency before and after migration. See relayrouter.io/docs for the exact base URL and key configuration, and confirm that streaming is enabled in your client so first-token timing can be captured.

How model choice affects first-token timing

Model selection influences TTFT because different models have different response characteristics across the two supported protocols. RelayRouter covers the Claude family, GPT-5.5, Gemini 3.5, plus DeepSeek, GLM, MiniMax and Moonshot (source relayrouter.io/models). When first-token latency matters, test several of these models under identical prompts and measure the first streamed token for each, then compare. Because both the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) endpoints are available, you can route the same benchmark payload through the protocol your SDK uses. Live per-model information is published at relayrouter.io/models, which you can consult when narrowing your candidate list.

Steps to benchmark and optimize TTFT

Follow an ordered process to compare latency across configurations.

1. Keep your existing SDK, then change base_url and the API key (no other code changes).
2. Enable streaming so the first token event can be timed.
3. Record a start timestamp, then capture the first streamed chunk timestamp to compute TTFT.
4. Repeat each run several times and average the results to reduce noise.
5. Swap in different models from the coverage list and re-run the identical prompt.

FAQ

Which protocols can I use to measure TTFT? Both the OpenAI protocol (/v1/chat/completions) and the Anthropic protocol (/v1/messages) are supported (source relayrouter.io/models), so you can measure using either ecosystem's streaming timers.

Do I need to rewrite my code to benchmark on RelayRouter? No. According to the official relayrouter.io/docs, you keep your existing SDK, change base_url and the key, with no other code changes.

Am I billed if a benchmark request fails? Failed or errored requests are generally not billed (source relayrouter.io), so failed timing runs do not incur charges.