BitRun ships a single binary that runs Microsoft BitNet b1.58 and other 1.58-bit ternary LLMs on macOS, Windows, and Linux CPUs without compilation steps, Docker images, or C++ toolchains. The project removes the friction that has kept 1-bit inference out of reach for most developers — recursive submodule clones, CMake configuration, manual AVX2 or NEON flag tuning, and 5–10 GB container images that defeat the low-power premise of ternary weights.

Why ternary models matter

Microsoft's BitNet b1.58 demonstrated that ternary models — weights constrained to -1, 0, and 1 — eliminate floating-point multiplications during inference. The result is 70 to 80 percent lower energy consumption on commodity CPUs compared to FP16 or INT4 models. These models target local CPU inference on battery-powered laptops, edge devices, and embedded workloads where GPU dependencies are impractical. The trade-off is model capacity: ternary models are typically smaller than their full-precision counterparts, but they deliver extreme token throughput for their size.

From source build to single binary

Testing BitNet.cpp upstream currently demands a full C++ development environment. Developers must clone recursive submodules, configure CMake, manage Python toolchains, and hand-tune compiler flags for their CPU's vector instruction set — AVX2, AVX-512, or ARM NEON. Many resort to heavy Docker images that consume the very power budget ternary models are designed to save. BitRun packages the native engine into a self-contained binary with zero external dependencies. Installation is a one-liner:

curl -fsSL https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh | sh

On Windows PowerShell:

irm https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1 | iex

Core workflows

After installation, models are pulled directly from Hugging Face and run immediately:

bitrun pull bitnet-2b
bitrun run bitnet-2b

The serve command starts an OpenAI-compatible REST server on port 11435, deliberately chosen to avoid the default Ollama port 11434. Any OpenAI-compatible client — Cursor, Continue.dev, Open WebUI, LangChain, Python SDK — points to http://localhost:11435/v1. A sample request:

curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bitnet-70m",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 0.7
  }'

Utility commands include bitrun hardware for CPU topology and vector instruction detection, bitrun list for downloaded models and custom personas, and one-shot prompts via bitrun run falcon3-1b "Explain RAII in C++".

Performance on real hardware

Benchmarks show prompt and generation throughput across three model sizes. On Apple Silicon and Linux x86_64 with SIMD acceleration:

  • Falcon3 1B Instruct (1.30 GB): 137–142 tok/s prompt, 98–127 tok/s generation, ~1.9 GB RAM
  • BitNet b1.58 2.4B (1.13 GB): 60.1 tok/s prompt, 39.8 tok/s generation, ~1.8 GB RAM
  • BitNet 70M (79 MB): 8,758 tok/s prompt, 1,355 tok/s generation, <150 MB RAM

On AVX2-capable x86_64, Falcon3 1B reaches 209.6 tok/s prompt and 39–45 tok/s generation; BitNet 70M hits 2,419 tok/s prompt and 492 tok/s generation. All figures report 0% GPU utilization — pure CPU SIMD.

Model catalog and custom personas

Four models are available in the built-in catalog:

  • falcon3-1b — 1B parameters, i2_s Ternary, ~1.30 GB, 4,096 context, TII Falcon3 1B Instruct
  • bitnet-2b — 2.4B parameters, i2_s Ternary, ~1.13 GB, 2,048 context, Official Microsoft BitNet b1.58 2B4T
  • falcon3-3b — 3B parameters, i2_s Ternary, ~2.80 GB, 4,096 context, TII Falcon3 3B Instruct
  • bitnet-70m — 70M parameters, Q8_0, ~79 MB, 768 context, Ultra-compact smoke test model

Custom personas are defined via Modelfiles using familiar syntax:

FROM falcon3-1b
SYSTEM You are an expert Go systems engineer. Write dense, production-grade code.
PARAMETER temperature 0.2
PARAMETER context_size 4096

Create and run the persona:

bitrun create gopher -f Modelfile
bitrun run gopher "Write a ring buffer implementation"

Architecture: supervisor and engine

BitRun uses a decoupled supervisor-engine design. The supervisor layer is written in Go and handles cross-platform CLI, CPU topology detection, a content-addressed asset store at ~/.bitrun, and the HTTP REST server. The engine tier is a statically compiled native C++ binary that uses ARM NEON with dot-product support (ARM_FEATURE_DOTPROD) and x86 AVX-512 kernels. Zero CGO dependencies mean the engine runs in total crash isolation from the supervisor.

Upstream lineage and contributions

The engine builds on two MIT-licensed foundations: llama.cpp by Georgi Gerganov and community (pioneering CPU inference architecture and GGUF specification), and BitNet.cpp by Microsoft Research (1-bit and 1.58-bit ternary matrix kernels). BitRun has already contributed fixes upstream: ARM64 2B batch stride, FFN activation, and LLaMA-3 tokenizer inference fixes (microsoft/BitNet#642), and native MSVC compiler support with CLI build tools on Windows (microsoft/BitNet#644).

What this changes for developers

BitRun makes ternary LLM inference accessible without the infrastructure tax. Teams can embed 1-bit models in local tools, CI pipelines, or edge deployments by pulling a binary and a model file — no compiler, no container runtime, no GPU. The OpenAI-compatible server drops into existing LLM integrations. Modelfiles give a portable way to ship task-specific personas. For battery-constrained or air-gapped environments, this is the first distribution path that matches the efficiency promise of ternary weights.