Show HN: Vlmhub – Simple Unified Python interface for vision-language models

vlmhub Config-Driven Vision-Language Model Inference

vlmhub is a lightweight framework for unified vision-language model inference across local and cloud backends. The system supports Ollama, MLX-VLM, vLLM, HuggingFace Transformers, Gemini, Vertex AI, OpenAI, and Anthropic without requiring rewritten inference code for each provider. A single Model.generate() call handles TextBlock and ImageBlock combinations regardless of the underlying model or backend.

One Interface Every Backend

The core proposition of vlmhub is a unified API. One Model.generate() call processes interleaved text and image blocks across any supported backend. The framework routes prompts to the appropriate provider, handling encoding differences such as base64 data URIs for API-hosted models or PIL in-process for Transformers. This means a single code path can target local servers, cloud APIs, or in-process model loading without conditional logic in the calling code.

Model Registry and Configuration

Models, settings, and generation defaults live in a single editable JSON file. The registry maps short names to real model identifiers and holds per-hosting configuration such as backend selection, API base URLs, and model-specific fields. Pre-configured entries include Gemma 3 (4B and 12B) and Qwen3-VL (4B and 8B) across all local hostings, plus Qwen2.5-VL, InternVL3.5, mPLUG-Owl3, Molmo2, Idefics3, and LLaVA-OneVision under the Transformers backend. Open-weight judges such as Qwen3.6/3.8, Prometheus, CompassJudger, UnifiedReward, and gpt-oss are available under vLLM, while cloud entries cover Gemini 2.5/3.x, GPT-4o/4.1/5.x, Claude 4.5+, and Gemma 4 (via Gemini). Users may extend the registry by providing a custom JSON file that deep-merges over the bundled default, adding hostings or overriding individual model fields.

Backend Support Matrix

Installation and Getting Started

The package is installed as an editable library. From the repository root, running pip install -e . installs the core package without PyTorch — the in-process Transformers backend is the only requirement for PyTorch. Additional extras pull in specific dependencies: .[transformers] adds torch and Transformers, .[quantization] adds bitsandbytes for 4-bit loads, .[ollama] manages Ollama's store, and .[all] installs everything. A submodule workflow is also supported for incorporating vlmhub into other projects as a git submodule under src/_libs/vlmhub.

After installation, the .env.example file provides placeholder variables for API keys and the HuggingFace token. Only the keys for intended backends need to be filled in — a purely local Ollama or vLLM setup requires none of them. Backend-specific setup commands include brew install ollama and ollama pull gemma3:4b followed by ollama serve for local development, or python -m mlx_vlm.server --model mlx-community/gemma-3-4b-it-qat-4bit --port 8080 for Apple Silicon, and pip install vllm followed by vllm serve Qwen/Qwen3-VL-4B-Instruct --port 8000 for CUDA GPUs.

Basic Usage Pattern

The public API imports Model, TextBlock, and ImageBlock. A model instance is created by passing a client identifier such as "ollama/gemma3-4b" or "gemini/flash-3.5". The generate() method accepts a list of content blocks — TextBlock and ImageBlock in any order. ImageBlock accepts an image_path for files on disk or an image= parameter for a PIL image in memory. The framework automatically encodes images as base64 data URIs for API-hosted backends or as PNG for Transformers in-process loading.

from vlmhub import Model, TextBlock, ImageBlock

model = Model("ollama/gemma3-4b")
reply = model.generate([
    TextBlock("Here are two images:"),
    ImageBlock(image_path="image1.png"),
    ImageBlock(image=pil_image),
    TextBlock("What are the main differences between them?"),
])
print(reply["text"])

Generation parameters originate from the registry defaults, can be overridden per-hosting, and further overridden per-call. For example, model.generate(content, max_new_tokens=1024, temperature=0.0, top_p=1.0, top_k=40, reasoning_effort="medium") passes those values to the backend. A None value is not sent, so the provider's default applies. Models marked "sampling": false in the registry — such as Claude Sonnet 5, Opus 4.8+, and GPT-5.x — reject temperature, top_p, and top_k; dropped values appear in logs with a DROPPED ... line, and the params property reflects what was actually sent. For reproducible runs, using dated model IDs where the provider offers them and recording the model and params with each result is recommended.

Model Properties and Image Handling

Useful properties include model.name (e.g. "ollama/gemma3-4b"), model.model_id (e.g. "gemma3:4b"), model.short_name (e.g. "gemma3-4b"), and model.report() to print both the model identifier and generation parameters. Images are encoded automatically — base64 data URI for every API-hosted backend, PIL for Transformers. An ImageBlock(image=...) is encoded as PNG, so nothing is re-compressed on the way out.

Registry Management

The model registry lives in src/vlmhub/models.json, which ships with the package. The file contains an active client mapping, default generation parameters, and hostings sections mapping each provider to its backend, API base, and model list. Editing models.json in place works for quick changes, but git submodule users should be aware that edits land in vlmhub's history rather than their project's, and a release touching models.json may block the next git submodule update until those changes are stashed or committed. For project-local extensions, a separate JSON file can be passed to Model() via the models_path parameter; it deep-merges over the bundled registry, allowing hostings and models to be added or single fields overridden. One caveat: lists in the registry are replaced, not appended, so naming an existing hosting's models drops the rest — adding a new hosting key is the correct approach.

Transformers-Specific Configuration

The in-process Transformers backend accepts per-model fields such as model_class, fallback_dtype, and an optional processor_kwargs dict. The fallback_dtype parameter targets pre-Ampere GPUs (e.g. V100) that lack native bfloat16 support. The registry also provides functions for HuggingFace cache and Ollama store management — list_hf_cache_models, download_hf_model, delete_hf_cache_model, and interactive pickers are available via the command python -m vlmhub.utils.local_models. Ollama functions require the ollama extra dependency.

Test Scripts and Verification

The repository ships test scripts in the tests/ directory that serve as both smoke tests and usage examples. These tasks download real dataset samples, run them through clients, and print output alongside ground truth. test_task_captioning.py uses COCO samples, test_task_vqa.py uses VQAv2 samples, test_clients.py checks a list of clients end to end, and test_judy.py grades built-in answers as an LLM judge on a 1-5 rubric. Running the tests from a clone requires the active client from models.json, with examples such as python tests/test_task_captioning.py, python tests/test_task_vqa.py --client ollama/gemma3-4b, and python tests/test_judge.py --client openai/gpt-4o-mini. A Slurm job script test_vllm.sh is provided for running tests against a vLLM-served model.

Extending the Framework

Adding a new provider that LiteLLM can reach requires only a new hosting entry in the registry naming the LiteLLM provider prefix — no code changes needed. For genuinely new backend types outside LiteLLM's reach, such as the in-process Transformers backend, subclassing BaseBackend in backends.py and registering it in get_backend_from_config is necessary. The framework is released under the MIT License.