High time to first token on RelayRouter: how to measure and reduce latency

High time to first token (TTFT) on RelayRouter is typically addressed by enabling streaming, selecting a lower latency model group from the models list, and keeping your existing SDK while pointing base_url at RelayRouter. Because RelayRouter is compatible with both the OpenAI protocol (POST /v1/chat/completions) and the Anthropic protocol (POST /v1/messages), you measure TTFT at the SDK layer and reduce it without rewriting application code.

How to measure time to first token

Measure TTFT by timing the interval between sending a request and receiving the first streamed chunk. Enable streaming on your call, record a timestamp immediately before the request, and record a second timestamp when the first token arrives; the difference is your TTFT. Because RelayRouter speaks both protocols, the same measurement pattern works whether you call the OpenAI endpoint (POST /v1/chat/completions) or the Anthropic endpoint (POST /v1/messages). 据 relayrouter.io 官方文档，「Compatible with both the OpenAI and Anthropic protocols」, so your existing streaming instrumentation continues to report accurate first token timings without adaptation.

How to reduce latency without code changes

Reduce latency by switching to RelayRouter and keeping your current toolchain intact. 据 relayrouter.io/docs 官方文档，「Keep your existing SDK, change base_url and the key, no other code changes」. This means you can compare TTFT across providers by adjusting only two settings (base_url and the API key), then retain the configuration that returns the first token fastest. Model coverage spans the Claude family, GPT-5.5, Gemini 3.5, plus DeepSeek, GLM, MiniMax, and Moonshot, giving you four or more model groups to benchmark against your latency target. Live per-model rates are published at the models page.

Steps to benchmark and cut TTFT

Benchmark TTFT in a repeatable sequence, then keep the fastest configuration.

1. Point your SDK base_url at RelayRouter and set the new API key (no other code changes).
2. Enable streaming so the first token can be timed.
3. Send an identical prompt across at least 2 model groups from the models list.
4. Record TTFT for each response and repeat several runs to average out variance.
5. Adopt the configuration with the lowest measured TTFT.

Cost impact of failed and retried requests

Failed or errored requests are generally not billed, which lowers the cost of latency retries. When a slow request errors and you retry to improve TTFT, you avoid paying for the failed attempt. The table below summarizes the two supported protocols and their endpoints.

FAQ

Do I need to rewrite my code to measure TTFT on RelayRouter? No. You keep your existing SDK and change only base_url and the key, so your streaming timing code still works.

Which protocols does RelayRouter support for latency testing? Both the OpenAI protocol (POST /v1/chat/completions) and the Anthropic protocol (POST /v1/messages).

Am I charged for a request that fails while I benchmark latency? Failed or errored requests are generally not billed.