Image captioning (VLM)

With the [vlm] extra, docparser sends every extracted image to a vision-language model along with its surrounding caption and context, and stores a structured caption back into the JSON. Captions are cached by image hash, so re-runs are free.

How it works#

Each image is sent to a VLM (default provider OpenRouter, model anthropic/claude-sonnet-4) together with its nearby caption and the context_before / context_after text captured during parsing. The model returns a strict JSON object:

{
  "caption":           "one-sentence figure caption",
  "description":       "2–5 sentence paragraph",
  "visible_text":      "OCR-style transcription",
  "tags":              ["world-model", "diagram", "..."],
  "image_kind":        "diagram | plot | screenshot | photo | equation | table | ...",
  "domain_relevance":  "how this relates to the document's topic"
}

Providers#

Any OpenAI-compatible provider works via --vlm-provider (or the library's provider= argument):

# OpenRouter (default)
export OPENROUTER_API_KEY=sk-or-v1-...
docparser parse-all --workspace ./project --max-images 50
 
# OpenAI
docparser parse-all --workspace ./project --vlm-provider openai --vlm-model gpt-4o-mini
 
# Fully local
docparser parse-all --workspace ./project --vlm-provider transformers

Caption caching#

Results are cached on disk at <cache_dir>/vlm/<model>/<sha1>.json, keyed by SHA-1 of the image bytes × model. Re-runs are free until the source image bytes change — and switching models produces a separate cache entry rather than overwriting.

Library usage#

from docparser import caption_image
 
result = caption_image(
    image_bytes,
    mime="image/png",
    doc_name="paper.pdf",
    nearby_caption="Figure 1: model architecture",
    context="...surrounding text...",
    provider="openrouter",
)

caption_image returns a VLMResult and requires the [vlm] extra.

Next#

Configure provider keys and defaults in Configuration.