FastAPI plus httpx: async calls to RelayRouter with concurrency limits

To call RelayRouter asynchronously from FastAPI, create one shared httpx.AsyncClient at startup with the base URL https://relayrouter.io/v1 and an Authorization: Bearer YOUR_API_KEY header. Send requests to POST /v1/chat/completions and wrap each call in an asyncio.Semaphore that caps how many requests run at once. The semaphore value is a client-side setting you choose, so the event loop stays unblocked while your app controls its own concurrency.

How do I configure httpx to talk to RelayRouter?

Point a single httpx.AsyncClient at https://relayrouter.io/v1 and attach your RelayRouter key as a Bearer token. RelayRouter accepts three request formats: OpenAI (/v1/chat/completions), Anthropic (/v1/messages, base https://relayrouter.io) and Gemini (/v1beta/models/{model}:generateContent). According to the official relayrouter.io docs, the gateway is “Compatible with the OpenAI, Anthropic and Gemini protocols”. With raw httpx you choose the path that matches the payload shape you already send. If you use an SDK instead, the official relayrouter.io/docs say: “Keep your existing SDK, change base_url and the key, no other code changes”. You create API keys at https://relayrouter.io/dashboard. Streaming is supported, so client.stream("POST", ...) works for token-by-token responses.

How do I add a concurrency limit in FastAPI?

Use an asyncio.Semaphore shared across requests, combined with one long-lived client. These steps set it up:

1. In a FastAPI lifespan handler, create httpx.AsyncClient(base_url="https://relayrouter.io/v1", headers={"Authorization": f"Bearer {KEY}"}, timeout=60) and store it on app.state.
2. Create sem = asyncio.Semaphore(N) at module level, where N is a value you choose for your workload.
3. In your route, run async with sem: and inside it call await client.post("/chat/completions", json={"model": "gpt-5.6-sol", "messages": msgs}).
4. Call r.raise_for_status() and return r.json().
5. Close the client with await client.aclose() when the lifespan handler exits.

For batch work, use asyncio.gather over many tasks; the semaphore keeps in-flight calls at N.

Which models can I call, and what does it cost?

You can call about 108 models across 19 public groups through the same client by changing the model field. The catalog at relayrouter.io/models includes the Claude family (claude-opus-5-5, claude-fable-5-1, claude-opus-5), GPT-6 and GPT-5.6 (gpt-6-astra, gpt-5.6-sol), Gemini 3.8 Flash (gemini-3.8-flash), plus DeepSeek, GLM, MiniMax and Moonshot. Settlement rates are CNY 0.6 per $1 of standard usage for the GPT group and CNY 2.0 per $1 for the Claude group, against a market reference of CNY 6.8 per $1. Direct pricing for deepseek-v4-flash is CNY 1.1 per 1M input tokens and CNY 4.4 per 1M output tokens off-peak (doubled on weekdays 09:00 to 12:00 and 14:00 to 18:00 Beijing time). There is a $0 platform fee, no minimum spend and no subscription. Payment is by Stripe card. Failed or errored requests are generally not billed, so retries after errors do not add charges for the failed attempts.

FAQ

Do I need a RelayRouter-specific SDK to use httpx?
No. httpx sends standard HTTP requests to https://relayrouter.io/v1/chat/completions with a Bearer header, using the OpenAI request format.

Is the semaphore value a RelayRouter limit?
No. It is a client-side cap you set in your FastAPI app. Check relayrouter.io/docs for any account-level details.

Can I switch from GPT to Claude without changing my httpx code?
Yes, when using the OpenAI-compatible endpoint you change the model value, for example to claude-opus-5-5. Live per-model rates are listed at relayrouter.io/models.