Providers

llm-rotate ships a built-in catalog of seven providers. Each has an auth method, a client adapter, and default cooldown/quarantine values you can override in configuration.

Supported providers#

Built-in defaults#

OpenRouter's known upstreams are anthropic, openai, google_ai_studio, and google_vertex.

Auth methods#

API-key providers#

openai, anthropic, google_ai_studio, and openrouter all use a single API key referenced via secret_ref:

{
    "key_id": "anthropic-1",
    "provider": "anthropic",
    "secret_ref": "env://ANTHROPIC_API_KEY",
    "models": ["claude-haiku-4-5-20251001"],
}

Google Vertex AI#

Vertex uses service-account credentials and requires project + region in auth_config. The service-account JSON is resolved through secret_ref (e.g. an env var holding the JSON), or you can rely on Application Default Credentials.

{
    "key_id": "vertex-1",
    "provider": "google_vertex",
    "secret_ref": "env://VERTEX_SA_JSON",
    "auth_method": "service_account",
    "auth_config": {"project": "my-gcp-project", "region": "us-central1"},
    "models": ["gemini-2.0-flash"],
}

Azure OpenAI#

Azure OpenAI is wire-compatible with the OpenAI Chat Completions API, so it rides on the core openai dependency — no extra to install. The model you pass is the Azure deployment name, and the endpoint comes from auth_config:

{
    "key_id": "azure-1",
    "provider": "azure_openai",
    "secret_ref": "env://AZURE_OPENAI_API_KEY",
    "auth_config": {
        "azure_endpoint": "https://my-resource.openai.azure.com",
        "api_version": "2024-10-21",
    },
    "models": ["my-gpt-4o-deployment"],
}

AWS Bedrock#

Bedrock uses the Converse API and the aioboto3 SigV4 stack (pip install "llm-rotate[bedrock]"). region is required in auth_config; credentials can come from an explicit access key, a named profile, or an instance role:

# Explicit access key — secret_ref resolves the AWS secret access key.
{
    "key_id": "bedrock-1",
    "provider": "aws_bedrock",
    "secret_ref": "env://AWS_SECRET_ACCESS_KEY",
    "auth_config": {
        "region": "us-east-1",
        "aws_access_key_id": "AKIA...",
    },
    "models": ["anthropic.claude-3-5-sonnet-20241022-v2:0"],
}
 
# Profile or instance role — use a literal:// placeholder for the unused secret.
{
    "key_id": "bedrock-role",
    "provider": "aws_bedrock",
    "secret_ref": "literal://unused",
    "auth_config": {"region": "us-east-1", "profile_name": "my-profile"},
    "models": ["anthropic.claude-3-5-sonnet-20241022-v2:0"],
}

Model-ID conventions#

The model string you pass to chat() must be valid for whichever provider serves it. This matters most for OpenRouter, which uses namespaced model IDs:

How the provider is resolved#

For each call, llm-rotate decides which provider's keys to try:

1. Explicit provider= argument → only that provider's keys are tried; the fallback chain is not consulted.
2. No provider= → the provider is inferred from the model name:
   • First, any active key whose models list contains the model.
   • Then the provider catalog's supported_models.
   • Then prefix heuristics: claude* → anthropic, gpt-* / o1- / o3- / o4- → openai, gemini* → google_ai_studio.
3. On exhaustion → the fallback_chains entry keyed by the inferred provider is walked in order.
4. Can't infer and no chain → ConfigurationError.