Research CommonsResearch Commons
llm-rotate/Fallback chains

Fallback chains

Survive a whole provider going down by transparently re-routing to another.

Key rotation handles a single failing credential. Fallback chains handle the bigger failure: every key for a provider is exhausted (rate-limited, quarantined, or auth-failed). When that happens, llm-rotate can re-try the same request against a different provider.

Configuring a chain

Pass fallback_chains to configure(). The keys are provider names; the values are ordered lists of fallback targets.

from llm_rotate import configure, lm
 
configure(
    registry={
        "keys": [
            {"key_id": "anthropic-1", "provider": "anthropic",
             "secret_ref": "env://ANTHROPIC_API_KEY",
             "models": ["claude-haiku-4-5-20251001"]},
            {"key_id": "openrouter-1", "provider": "openrouter",
             "secret_ref": "env://OPENROUTER_API_KEY",
             "models": ["anthropic/claude-haiku-4.5"]},
        ]
    },
    use_keys=["anthropic-1", "openrouter-1"],
    fallback_chains={
        "anthropic": [{"provider": "openrouter"}],
        "openai":    [{"provider": "openrouter"}],
    },
)
 
# Provider inferred from the model → on exhaustion, the chain under
# "anthropic" is walked in order.
response = await lm.chat(
    "claude-haiku-4-5-20251001",
    [{"role": "user", "content": "Hello"}],
)

Chain entry format

Each entry is an object:

{"provider": "openrouter", "upstream": "anthropic"}
  • provider (required) — the fallback provider to try next.
  • upstream (optional) — used by aggregators like OpenRouter to pin which backend the broker should route to.

When chains are (and aren't) used

SituationChain consulted?
provider= passed explicitlyNo — only that provider's keys are tried.
Provider inferred from model, keys exhaustedYes — chain for the inferred provider is walked.
Provider can't be inferred and no chainNo — raises ConfigurationError.
Match the model ID to the fallback provider

The model string must be valid on the fallback provider. A request for claude-haiku-4-5-20251001 that fails over to OpenRouter needs the OpenRouter model ID (anthropic/claude-haiku-4.5). llm-rotate does not rewrite model IDs for you — make sure each fallback key's models list contains a compatible ID.

Broker-aware routing

OpenRouter is a broker: it fronts multiple upstreams. llm-rotate tracks route health per upstream, so if OpenRouter→Anthropic is failing but OpenRouter→OpenAI is healthy, selection accounts for that. Use upstream in the chain entry to pin a specific backend when you need deterministic routing.

Exhausting the chain

If the native provider and every chain entry are exhausted, the call raises NoAvailableKeyError. In an HTTP service this is the natural mapping to a 503.