Research CommonsResearch Commons
llm-rotate/Advanced

Advanced

Secret resolution, the low-level acquire() escape hatch, runtime reload, and singleton lifecycle.

This page covers the less-common but useful corners of the API.

Secret resolution

secret_ref values are resolved at call time by resolve_secret(). Three schemes are supported:

SchemeExampleBehavior
env://env://OPENAI_API_KEYRead from os.environ.
file://file:///run/secrets/openaiRead the file, strip whitespace.
literal://literal://sk-...Use the value directly (logs a warning).
{"key_id": "k1", "provider": "openai",
 "secret_ref": "file:///run/secrets/openai_key",
 "models": ["gpt-4o-mini"]}
Avoid literal://

literal:// puts a secret straight into your config and is logged with a warning. Prefer env:// or file:// so credentials stay out of source and serialized configs. secret_ref is always excluded from model dumps.

The acquire() escape hatch

When you need to call a vendor SDK directly but still want llm-rotate's key selection and health tracking, borrow a key with acquire():

from llm_rotate import lm
 
async with lm.acquire("openai", model="gpt-4o-mini") as ctx:
    # ctx.key_value is the raw resolved secret — use any SDK you like.
    client = SomeOpenAIClient(api_key=ctx.key_value)
    try:
        result = await client.do_something()
        await ctx.report_success()
    except Exception as exc:
        await ctx.report_error(exc)
        raise

AcquireContext exposes key_value, provider, key_id, and metadata. Reporting success/error feeds the same health state machine the managed chat() path uses, so manual calls participate in rotation.

Runtime config reload

Swap configuration on a live instance without recreating it. Useful for rotating the key pool or adjusting strategy without a restart:

new_config = configure_from_dict(registry={"keys": [...]}, use_keys=[...])
rot.reload_config(new_config)

Distributed state

The default in-memory store keeps key health, cooldowns, and leases in-process. That's perfect for a single worker, but multiple workers each get their own view and can hammer the same key. The Redis backend (pip install "llm-rotate[redis]") fixes that by sharing state across every process pointed at the same Redis:

config = configure_from_dict(
    registry={
        "state_store": {"backend": "redis", "redis_url": "redis://localhost:6379/0"},
        "keys": [...],
    },
    use_keys=[...],
)
rot = LMRotate(config)

What is and isn't shared:

  • Shared: key health, cooldown/quarantine timers, and advisory leases — so rotation and rate-limit backoff coordinate across workers.
  • Process-local: the key→provider mapping (it's static config each worker learns from its own config) and the usage event buffer.

The namespace (redis_namespace, default llmrotate) prefixes every key, so several apps can safely share one Redis instance. await rot.close() releases the Redis connection. See Configuration for all fields.

Singleton lifecycle

  • configure() initialises the process-wide lm. It's an error to call it twice.
  • reset_singleton() (importable from llm_rotate) tears the singleton down — intended for tests so each test can configure fresh.
  • await rot.close() releases resources held by a direct LMRotate instance.
import llm_rotate
 
def teardown():
    llm_rotate.reset_singleton()