A new open-source CLI tool called VeloxML aims to remove the friction from deploying open-source large language models into your own cloud infrastructure. The tool provisions optimized instances on AWS or GCP with a single command, eliminating the need to manage Docker containers, Kubernetes clusters, or third-party inference platforms.

One Command, Your Own Cloud

VeloxML reads a Python file and a YAML spec, then provisions a Spot or On-Demand instance directly inside your AWS or GCP account using SkyPilot, an open-source compute orchestrator from UC Berkeley. It installs dependencies, loads model weights, verifies the health probe, and prints a ready-to-use endpoint URL. The entire workflow from init to live endpoint takes under two minutes for small models like Qwen2.5-0.5B-Instruct running on a Spot instance at roughly $0.07 per hour.

The core pitch is straightforward: data, prompts, and weights never leave your own account. That sidesteps SOC2 and HIPAA compliance reviews that come with third-party inference services. And unlike Modal, which requires rewriting code with proprietary decorators, VeloxML works with standard FastAPI and Transformers code you already have.

What the Setup Looks Like

A minimal deployment requires two files. The Python file defines a FastAPI app with a health check and a predict endpoint. The YAML spec declares hardware requirements and runtime setup:

name: llm-service
compute:
  cpus: 4+
  memory: 8+
  use_spot: true
runtime:
  setup: pip install fastapi uvicorn "transformers<5.0.0" accelerate

Running veloxml deploy provisions the instance and returns an IP address. A simple curl test confirms the endpoint is live:

curl -X POST http://<ENDPOINT_IP>:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Say this is a test"}'

Use Cases and Current State

The tool targets private inference APIs for open-weights checkpoints or fine-tuned models, custom embedding and reranking services, real-time agent tool-calling backends, and high-throughput batch inference. VeloxML defaults to Spot instances, which means deployments can be interrupted, but the cost efficiency is significant compared to running idle managed GPU instances at $1,500 or more per month.

The project is still in alpha and under active development. The maintainers note that breaking changes are likely and recommend watching the releases feed for beta updates. It is licensed under Apache 2.0, and the implementation depends heavily on SkyPilot and the Truss model packaging framework.

The Broader Pattern

VeloxML fits into a growing category of tools that try to make self-hosted LLM deployment feel like serverless computing. The value proposition is clear for teams that need to keep sensitive data behind their own perimeter but do not want to build and maintain the underlying infrastructure. Whether VeloxML can handle production-grade workloads at scale remains to be seen, but the alpha already demonstrates the core workflow works for lightweight models on commodity hardware.