Agent Runtime

Cloudflare AI Gateway Free LLM Fallback Chain

Route coding agents through a Cloudflare-hosted OpenAI-compatible gateway backed by verified free or free-credit LLM providers, with BYOK keys, model aliases, dynamic fallback, and a no-key reserve route.

What This Builds

This recipe builds a single OpenAI-compatible endpoint on Cloudflare that coding agents can use as their model gateway:

  • Claude Code, Codex, Hermes Agent, OpenClaw, or any client that can set an OpenAI baseURL.
  • One Cloudflare AI Gateway in front for authentication, observability, logs, caching, rate limits, and model routing.
  • A fallback chain built from current strong free, free-credit, or already-keyed providers.
  • Bring Your Own Keys (BYOK), so application clients do not carry provider keys directly.
  • Model aliases such as dynamic/free-code for agents that should not know the provider list.

The current practical chain avoids older Llama 3.x defaults and weak Flash-era fallbacks. It uses the strongest models that were visible in the live provider catalogs and practical enough for an agent gateway.

RankRoleProviderDefault modelWhy it is hereCurrent status
1Primary frontier modelNVIDIA NIMmoonshotai/kimi-k2.6Current 1T-class Kimi model for long-horizon coding, tool use, and agentic workloads.Verified live via direct API.
2NVIDIA backupNVIDIA NIMdeepseek-ai/deepseek-v4-proStrong current DeepSeek coding/agent model in the same NVIDIA catalog.Verified live via direct API.
3Mistral flagship fallbackMistral AImistral-large-latestCurrent Mistral flagship endpoint; avoids older Devstral as the default.Verified live via direct API.
4Cloudflare-native code reserveWorkers AI@cf/moonshotai/kimi-k2.7-codeCurrent Cloudflare-hosted Kimi code model, useful when external provider keys throttle.Verified live via Workers AI.
5Coding reserveOllama Cloudqwen3-coder:480bStrong coder model available through the keyed Ollama Cloud endpoint.Verified live via OpenAI-compatible API.
6Emergency no-key reserveKilo Code Gatewaynex-agi/nex-n2-pro:freeLast-resort no-key route only; Kilo’s Kimi requires sign-in and Nemotron Ultra free timed out in smoke tests.Verified live; do not treat as primary quality capacity.

OpenRouter is intentionally not in the default chain. Without a funded balance it is a low-priority emergency fallback because the free daily request cap is too small for agent workflows.

Google Gemini is also not in the default free chain right now. The current key can list newer Gemini models, but Pro-class generation hit free-tier quota/billing limits during smoke tests. Use gemini-3.1-pro-preview only in a separate quality route after billing/quota is confirmed; do not fall back to older Flash-generation models.

Can The Gateway Route By Requested Model?

Yes, but there are three distinct mechanisms.

NeedCloudflare mechanismExample
Client chooses an exact provider/modelOpenAI-compatible model stringcustom-nvidia/moonshotai/kimi-k2.6, mistral/mistral-large-latest, workers-ai/@cf/moonshotai/kimi-k2.7-code, custom-ollama/qwen3-coder:480b
Client asks for an alias and Cloudflare chooses the providerDynamic Routedynamic/free-code
Same provider has multiple keys or accountsBYOK aliasesHeader cf-aig-byok-alias: production or cf-aig-byok-alias: free-trial

Use direct provider/model strings when the caller should control the exact model. Use Dynamic Routes when the caller should ask for a capability, for example dynamic/free-code, and let Cloudflare evaluate limits, failures, and fallbacks.

Cloudflare Setup

1. Create The Correct Cloudflare Tokens

The Workers AI token is not enough. A token that can call Workers AI can still return 403 or 401 on AI Gateway endpoints.

Create two tokens:

TokenRequired permissionsUsed for
Admin setup tokenAI Gateway - Edit; keep existing Workers AI permissions if this token also tests Workers AI.Creating custom providers, provider keys, dynamic routes, settings.
Runtime tokenAI Gateway - RunAgent requests through the gateway.

Cloudflare AI Gateway permissions are account-scoped, not per-gateway scoped. Treat a runtime token as sensitive because it can call any authenticated gateway in the account.

2. Create Or Use A Gateway

Use free-llm-agents as the gateway id unless the account already has a preferred gateway.

Dashboard path:

  1. Cloudflare Dashboard.
  2. Select the account.
  3. Go to AI > AI Gateway.
  4. Create gateway free-llm-agents.
  5. Open Settings and enable Authenticated Gateway.
  6. Save the runtime token as CLOUDFLARE_AI_GATEWAY_RUN_TOKEN.

Validation:

curl -sS \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai-gateway/gateways" \
  | jq '{success, errors, gateways: (.result | length)}'

3. Add Provider Keys With BYOK

Dashboard path:

  1. Open gateway free-llm-agents.
  2. Go to Provider Keys.
  3. Add keys for mistral, google-ai-studio, and any other native provider you want Cloudflare to manage.
  4. Use alias default for the normal chain.
  5. Use aliases like trial, paid, or backup only when there are multiple keys for the same provider.

For native BYOK requests, clients should send only the Cloudflare runtime token. Do not send provider Authorization headers from application clients.

4. Add Custom Providers

NVIDIA, Ollama Cloud, and Kilo are OpenAI-compatible enough to use through Cloudflare Custom Providers.

Custom provider slugBase URLNotes
nvidiahttps://integrate.api.nvidia.comUse provider-specific path /custom-nvidia/v1 with OpenAI SDKs.
ollamahttps://ollama.comUse provider-specific path /custom-ollama/v1.
kilohttps://api.kilo.ai/api/gatewayUse provider-specific path /custom-kilo or unified model custom-kilo/<model> if the dashboard accepts the fixed prefix.

API example after the setup token has AI Gateway - Edit:

curl -X POST \
  "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai-gateway/custom-providers" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "NVIDIA NIM",
    "slug": "nvidia",
    "base_url": "https://integrate.api.nvidia.com",
    "description": "NVIDIA NIM OpenAI-compatible hosted inference",
    "enable": true
  }'

Repeat for ollama and kilo.

Dynamic Route

Create a Dynamic Route named free-code. The route should attempt providers in this order:

  1. custom-nvidia/moonshotai/kimi-k2.6
  2. custom-nvidia/deepseek-ai/deepseek-v4-pro
  3. mistral/mistral-large-latest
  4. workers-ai/@cf/moonshotai/kimi-k2.7-code
  5. custom-ollama/qwen3-coder:480b
  6. custom-kilo/nex-agi/nex-n2-pro:free

Use short timeouts on the first two nodes so agent loops do not stall:

NodeTimeoutRetriesFallback
NVIDIA Kimi35s1NVIDIA DeepSeek
NVIDIA DeepSeek35s1Mistral
Mistral Large25s1Workers AI Kimi
Workers AI Kimi30s1Ollama Qwen Coder
Ollama Qwen Coder35s1Kilo
Kilo no-key30s0End with error

Call it from any OpenAI-compatible client:

curl -X POST \
  "https://gateway.ai.cloudflare.com/v1/$CLOUDFLARE_ACCOUNT_ID/free-llm-agents/compat/chat/completions" \
  -H "cf-aig-authorization: Bearer $CLOUDFLARE_AI_GATEWAY_RUN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "dynamic/free-code",
    "messages": [{"role": "user", "content": "Reply OK only."}],
    "max_tokens": 32
  }'

When a request succeeds, inspect these response headers:

  • cf-aig-provider
  • cf-aig-model

Those headers show which fallback branch actually handled the request.

Direct Model Calls

For debugging, call exact models before enabling the route:

# Native Mistral through Cloudflare AI Gateway.
curl -X POST \
  "https://gateway.ai.cloudflare.com/v1/$CLOUDFLARE_ACCOUNT_ID/free-llm-agents/compat/chat/completions" \
  -H "cf-aig-authorization: Bearer $CLOUDFLARE_AI_GATEWAY_RUN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral/mistral-large-latest",
    "messages": [{"role": "user", "content": "Reply OK only."}],
    "max_tokens": 32
  }'
# Custom NVIDIA through provider-specific endpoint.
curl -X POST \
  "https://gateway.ai.cloudflare.com/v1/$CLOUDFLARE_ACCOUNT_ID/free-llm-agents/custom-nvidia/v1/chat/completions" \
  -H "cf-aig-authorization: Bearer $CLOUDFLARE_AI_GATEWAY_RUN_TOKEN" \
  -H "Authorization: Bearer $NVIDIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/kimi-k2.6",
    "messages": [{"role": "user", "content": "Reply OK only."}],
    "max_tokens": 32
  }'

After BYOK is configured for the custom provider, remove the provider Authorization header and rely on Cloudflare’s stored key.

Client Configuration

For Codex, Hermes, OpenClaw, or any OpenAI SDK-compatible client:

OPENAI_BASE_URL=https://gateway.ai.cloudflare.com/v1/<account-id>/free-llm-agents/compat
OPENAI_API_KEY=<cloudflare-ai-gateway-run-token>
OPENAI_MODEL=dynamic/free-code

For clients that do not support a separate cf-aig-authorization header, use the newer Cloudflare REST API form instead:

OPENAI_BASE_URL=https://api.cloudflare.com/client/v4/accounts/<account-id>/ai/v1
OPENAI_API_KEY=<cloudflare-ai-gateway-run-token>
OPENAI_MODEL=dynamic/free-code

Operational Rules

  • Keep OpenRouter outside the default chain unless the account has at least a small funded balance.
  • Do not send sensitive prompts to Kilo no-key routes. Treat them as public-prototype capacity.
  • Keep Workers AI in the chain even if it is weaker than NVIDIA/Mistral/Gemini because it is the best same-account reserve.
  • Put hard per-user rate limits on dynamic/free-code before attaching it to autonomous agents.
  • Keep a manual quality route for slower premium-quality models such as NVIDIA nvidia/nemotron-3-ultra-550b-a55b and Google gemini-3.1-pro-preview; do not put them in the live free chain until latency and quota are confirmed.
  • Re-test provider availability weekly; free tiers and model catalogs drift.

Current Blocker

The current Cloudflare token in the local key store can call Workers AI, but does not have AI Gateway rights. It returns:

  • 403 Authentication error for accounts/<id>/ai-gateway/custom-providers
  • 403 Authentication error for accounts/<id>/ai-gateway/gateways
  • 401 Unauthorized when calling gateway.ai.cloudflare.com/.../compat

Create a new Cloudflare token with AI Gateway - Edit for setup and an AI Gateway runtime token with AI Gateway - Run. After that, this recipe is ready to apply.

Sources