Metrics & monitoring

gpu-train gives you two ways to see training metrics:

• Native, local-first — call gpu_train.log({...}) from your script and the values chart directly on the local dashboard. Works on every provider (including local), fully offline, no W&B account required.
• Weights & Biases (optional overlay) — when configured, runs are tracked automatically and the same gpu_train.log() call also forwards to wandb.log(). You can also pull a run's history back into the local dashboard on demand. The native views are complete on their own; W&B affordances only appear once a run has been tracked.

Native metrics: gpu_train.log()#

Call it from inside your training script:

import gpu_train
 
for step in range(steps):
    loss, acc = train_one_step()
    gpu_train.log({"loss": loss, "acc": acc}, step=step)

log(
    metrics: dict[str, float],
    *,
    step: int | None = None,
    all_ranks: bool = False,
) -> None

• metrics — a mapping of name → number. Booleans become 0/1; non-numeric values (and NaN/inf) are skipped.
• step — optional integer x-axis. When omitted, points are ordered by arrival.
• all_ranks — by default only rank 0 emits (reads RANK/LOCAL_RANK) so DDP/torchrun jobs don't duplicate points. Set True to log from every rank.

How it works#

gpu_train.log() prints a single sentinel-prefixed JSON line to stdout. That line rides the existing log stream back to the control plane, which parses it into a local metrics SQLite table (kept out of the raw log view) and serves it to the dashboard, where each key is drawn as a native SVG chart on the run detail page. No extra network path, no extra services.

Namespaced metrics & the grouped view#

Name your metrics with a namespace/stat convention and the dashboard groups them into collapsible sections instead of a flat wall of charts:

gpu_train.log({
    "train/loss": loss,
    "train/lr": lr,
    "eval/acc": acc,
    "perf/throughput": toks_per_s,
}, step=step)

Sections are ordered train, eval, stability, performance, then the rest alphabetically (mirroring prime-rl's curated overview). Keys without a / land in a default group.

Chart controls#

The Metrics panel has controls that apply to every chart at once:

• X-axis: step vs. time — plot against the logged step or the wall-clock timestamp (each point carries a ts).
• Smoothing — an EMA slider; the raw series is drawn faint behind the smoothed line.
• Log-scale — toggle a logarithmic Y-axis for losses that span orders of magnitude.
• Hover crosshair — hover any chart for a readout of the value and step under the cursor.

These are pure client-side SVG — no extra dependency, and they work fully offline.

Weights & Biases#

gpu-train integrates with Weights & Biases so every run is tracked automatically — you don't add any wandb glue beyond the usual wandb.init() / logging.

Enable it#

Add a tracking.wandb block to your registry:

configure(
    registry={
        "credentials": [...],
        "tracking": {
            "wandb": {
                "secret_ref": "env://WANDB_API_KEY",
                "project": "my-proj",
                "entity": "my-team",   # optional
                "enabled": True,
            }
        },
    },
    use=[...],
)

Or set WANDB_API_KEY (and optional WANDB_PROJECT / WANDB_ENTITY) in the environment, or connect W&B from the dashboard.

What happens#

When tracking is enabled, for each job the control plane:

1. Mints a stable W&B run id / group for the job.
2. Injects WANDB_API_KEY, WANDB_PROJECT, WANDB_ENTITY, and the run id into the remote job's environment (secrets written to owner-only files on the box).
3. Records the resulting wandb_run_id and wandb_url on the JobRecord, so the dashboard's run detail can deep-link straight into the W&B run.

Your train.py just calls wandb.init() as usual — it picks up the injected environment automatically and lands in the right project/run.

One call, both sinks#

When W&B is configured and a run is active, gpu_train.log() also calls wandb.log() with the same values. So a single gpu_train.log({...}) line feeds both the local dashboard and W&B — no double bookkeeping.

Load a run's history back into the dashboard#

For runs logged with plain wandb.log, the run detail page has a Load from W&B button. It calls GET /jobs/{id}/metrics/wandb, which reads the tracked run's history via the W&B Public API (run.history()) and overlays the series on the local charts — so you can skim W&B metrics without leaving the local view.

In the dashboard#

The Overview shows W&B status (connected / needs key); each run's detail view renders the Metrics panel (native charts from gpu_train.log), a Load from W&B button, and a deep-link into the W&B run. You can set or update the W&B key directly from the Providers page — see the Dashboard.