Running LLM tests in GitHub Actions CI against RelayRouter

To run LLM tests in GitHub Actions CI against RelayRouter, store a RelayRouter API key as an encrypted repository secret, expose it to your test job as an environment variable, and point your existing OpenAI, Anthropic or Gemini SDK at a RelayRouter base URL (for example https://relayrouter.io/v1 for OpenAI-compatible calls). Requests authenticate with Authorization: Bearer YOUR_API_KEY, and your test code needs no other changes.

How do you configure a GitHub Actions workflow to call RelayRouter?

You configure the workflow by saving the RelayRouter key as a secret and passing it, together with the base URL, into the environment of the test step.

1. Create an API key at https://relayrouter.io/dashboard.
2. In your repository, open Settings, then Secrets and variables, then Actions, and add a secret such as RELAYROUTER_API_KEY.
3. In the job, map the secret into the variable your SDK reads, for example OPENAI_API_KEY: ${{ secrets.RELAYROUTER_API_KEY }}, and set the base URL to https://relayrouter.io/v1 (or pass both values to your client constructor).
4. Run your existing test command, such as pytest or npm test.

According to the official relayrouter.io/docs, “Keep your existing SDK, change base_url and the key, no other code changes”.

Which API protocol should your CI tests use?

Your tests should use the protocol that matches the SDK your application already calls, because RelayRouter accepts all three major formats. According to the official relayrouter.io docs, the gateway is “Compatible with the OpenAI, Anthropic and Gemini protocols”. The endpoints are:

• OpenAI compatible: POST /v1/chat/completions, base https://relayrouter.io/v1
• Anthropic compatible: POST /v1/messages, base https://relayrouter.io
• Gemini compatible: POST /v1beta/models/{model}:generateContent

Streaming is supported, so tests that assert on streamed output can run against the same gateway. Every protocol uses the same bearer token header, which means one CI secret can serve test suites written for different SDKs within a single repository.

Which models can a CI test suite target?

A CI test suite can target any of about 108 models across 19 public groups listed at relayrouter.io/models. Coverage includes the Claude line (claude-opus-5-5, claude-fable-5-1 and claude-opus-5), GPT-6 and GPT-5.6 (gpt-6-astra, gpt-5.6-sol), Gemini 3.8 Flash (gemini-3.8-flash), plus DeepSeek, GLM, MiniMax and Moonshot. A practical pattern is to read the model ID from an environment variable and use a GitHub Actions matrix to run the same tests against several models in parallel jobs. Because all of these models sit behind one base URL and one key, adding a model to the matrix is a configuration change rather than a code change, and regression checks across vendors can share the same assertions.

How much do LLM test runs in CI cost on RelayRouter?

CI test runs are billed per usage with a $0 platform fee, no minimum spend and no subscription. Per the relayrouter.io/models page, the GPT group settles at CNY 0.6 per $1 of standard usage and the Claude group at CNY 2.0 per $1, against a market reference of CNY 6.8 per $1. Some models use direct pricing: deepseek-v4-flash costs CNY 1.1 per 1M input tokens and CNY 4.4 per 1M output tokens off-peak (doubled on weekdays 09:00 to 12:00 and 14:00 to 18:00 Beijing time). Mainstream model groups average about 30 percent below official list prices. Failed or errored requests are generally not billed, which matters for CI suites that retry or exercise error paths. Payment is by Stripe card, and live per-model rates are published on the models page.

FAQ

Do I need to rewrite my tests to use RelayRouter in CI?
No. You keep your existing OpenAI, Anthropic or Gemini SDK and change the base URL and API key; no other code changes are required.

How should the API key be stored in GitHub Actions?
Store it as an encrypted repository secret, then map it into an environment variable in the workflow job so it is never committed to the codebase. The key is sent as Authorization: Bearer YOUR_API_KEY.

Will failing test requests add to my bill?
Failed or errored requests are generally not billed, and there is a $0 platform fee with no minimum spend, so you pay for successful usage at the rates listed on relayrouter.io/models.