Batchlane is a Python library that gives developers a single interface for submitting asynchronous batch jobs across multiple LLM providers, each of which offers its own discounted pricing lane. The project, hosted on GitHub under the gojiplus organization, is built around a simple premise: abstract away the differences between provider batch APIs so a developer can focus on the work rather than the wiring.

The problem batchlane solves

Every major LLM provider offers a batch API with significant discounts compared to synchronous endpoints — typically 50% off the standard rate. But each provider implements that batch interface differently. OpenAI expects file uploads and a 24-hour completion window. Anthropic processes inline requests with no file upload step. Groq supports both 24-hour and 7-day windows. Gemini switches between inline requests and keyed JSONL file input depending on batch size, with a 2GB provider file limit. Fireworks requires dataset uploads and offers no cancel endpoint.

Writing separate code for each of these is tedious and error-prone. Batchlane consolidates that complexity into one Python module. A developer writes a list of prompts and a model identifier, and the library handles chunking to fit provider caps, submitting each batch, polling until completion, and rejoining results back to the original rows in input order.

How it works in practice

The core API is concise. A few lines of code map a model to a list of prompts and return answers:

import batchlane as bl
model = "groq/llama-3.3-70b-versatile"
prompts = ["The product was great.", "It broke in a week."]
answers = bl.map(model, prompts, system="Classify the sentiment.")

Behind the scenes, batchlane splits the job across whatever chunks the provider requires, submits them, and waits. Results come back in the original order, with None in place of any row that failed. The library also supports a file-to-file workflow from the command line, a checkpoint-based resumption system for long-running jobs, and a per-row control API via the run() function.

For developers who need a more familiar interface, batchlane exposes an OpenAI-compatible gateway. Running batchlane serve starts a local server at localhost:8000 that speaks standard OpenAI Batch API. The interesting detail is that the underlying model identifiers use provider-prefixed names like groq/llama-3.3-70b-versatile or gemini/... — so the batch runs against a provider the OpenAI client has never heard of. Any language or tool that speaks the OpenAI protocol can drive it.

Coverage and its limits

Eight providers are supported: Anthropic, Gemini AI Studio, OpenAI, Groq, Mistral, Fireworks, Together, and DeepInfra. Each offers a different discount percentage, result retention window, and set of constraints. The discount table is straightforward — most providers offer 50% off, though DeepInfra sits at 20% and Together varies by model. The Windows range from immediate (Anthropic) to 24 hours, 7 days, or anything in between (Mistral).

An honest constraint: only Anthropic has been verified end-to-end against a live API. The other adapters have passed mocked contract tests — their wire formats have been checked line by line against each provider's own API reference, which is not the same as proving they work in production.

Design decisions worth noting

Batchlane is deliberately strict about what it will and will not do. It raises NoBatchLaneError for providers classified as having no batch lane, and AdapterNotShippedError for adapters that simply have not been written yet. A provider whose lane exists but is unimplemented gets a different error. This means a refusal never claims a lane is absent when it is merely unwritten.

The library also refuses to serve as a substitute for batch APIs by sending concurrent synchronous requests. Self-hosted runtimes like Ollama, LM Studio, and vLLM are explicitly excluded because there is no per-token discount to apply — the hardware is already yours. For vLLM specifically, batchlane names the command vllm run-batch as the alternative path.

Gemini carries a specific hazard that the library documents plainly. Google's own API maps inline results by array index rather than by the custom key you supply. Batchlane joins on an echoed key wherever the payload carries one, falls back to submission order otherwise, and refuses outright when the counts disagree. This is a sensible safeguard: a silently mis-joined batch attaches plausible answers to the wrong rows, and nothing in the output looks wrong.

What this means for developers

For teams running large-scale inference — whether for data labeling, content generation, classification pipelines, or evaluation — batchlane removes a layer of provider-specific plumbing. The checkpoint-based resumability is particularly useful for long jobs: a recorded handle reattaches to a submitted job on retry, and changed requests are rejected rather than silently duplicated.

The serve mode opens a practical door. A developer can point an OpenAI-compatible client at a local batchlane gateway and have it route jobs to Groq, Gemini, or Anthropic without changing their client code. That is a genuinely useful abstraction, and the fact that the server stores no jobs — relying on the stateless nature of the OpenAI batch protocol — means there is no database to manage or migrate.

At pip install batchlane and Python 3.11, the barrier to entry is low. The MIT license keeps it unrestricted. The main caveat is that batch APIs are generally excluded from free tiers, and inference costs real compute regardless of who is billing — so batchlane is a tool for production workloads, not experimentation.