LlamaIndex with RelayRouter: pointing the LLM layer at an OpenAI compatible gateway

To use LlamaIndex with RelayRouter, keep the OpenAI LLM class you already have, then set the base URL to the RelayRouter OpenAI compatible endpoint (POST /v1/chat/completions) and swap in your RelayRouter API key. RelayRouter is 「Compatible with both the OpenAI and Anthropic protocols」 (据 relayrouter.io 官方文档), so no other code changes are required. You then select any covered model, from the Claude family to GPT-5.5 and Gemini 3.5.

How the OpenAI compatible endpoint fits LlamaIndex

LlamaIndex talks to RelayRouter through the OpenAI compatible route at /v1/chat/completions, which is the same interface LlamaIndex already expects. Because RelayRouter supports both the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) protocols, your existing LlamaIndex OpenAI integration works once the base URL points at the gateway. According to the official relayrouter.io/docs guidance: 「Keep your existing SDK, change base_url and the key, no other code changes」 (据 relayrouter.io/docs 官方文档). This means the LlamaIndex query engine, retrievers and agents continue calling the LLM layer as before, with the only differences being the endpoint and the API key you provide.

Migration steps for the LLM layer

Migrating the LlamaIndex LLM layer requires two edits: the base URL and the API key.

1. Locate where you construct the OpenAI LLM object in your LlamaIndex setup.
2. Set the base_url (or api_base) to RelayRouter's OpenAI compatible endpoint that serves /v1/chat/completions.
3. Replace the API key value with your RelayRouter key.
4. Choose a model name from the covered set (see relayrouter.io/models).
5. Run your existing LlamaIndex pipeline without further edits.

These are the only two required changes because RelayRouter mirrors the OpenAI protocol. Your indexes, embeddings configuration and prompt templates stay untouched, and the request and response shapes remain identical to a direct OpenAI call.

Model coverage available to LlamaIndex

Through one gateway, LlamaIndex can reach several model families without separate integrations. RelayRouter's model coverage spans the Claude family, GPT-5.5 and Gemini 3.5, plus DeepSeek, GLM, MiniMax and Moonshot. Because all of these are reachable over the same OpenAI compatible interface, you can switch the model name in your LlamaIndex configuration and route the same query engine to a different provider. Live per-model rates are published at relayrouter.io/models. This lets you evaluate multiple models for retrieval augmented generation or agent workflows while keeping a single LlamaIndex codebase and a single API key.

Protocol and billing summary

The table below compares the two protocols RelayRouter exposes and a billing detail relevant to LlamaIndex experimentation.

For LlamaIndex users iterating on prompts and retrieval settings, the fact that failed or errored requests are generally not billed reduces the cost of debugging runs that error out during development.

FAQ

Do I need to change my LlamaIndex code beyond the LLM setup? No. You change the base URL and the key; no other code changes are required, and your indexes and prompts stay the same.

Can I use the Anthropic protocol instead? Yes. RelayRouter supports both the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) protocols.

Am I billed if a request fails? Failed or errored requests are generally not billed. See relayrouter.io/models for live per-model rates.