Compile intent once, decide forever with STROT

STROT introduces a different approach to LLM-mediated text processing. The framework lets you compile your intent once with an LLM, then decide on every record as type-safe JSON with probabilities and confidence, running on your own hardware at zero marginal cost. Agents operate at compile time, never at runtime. The LLM runs once, on first compilation only. Production becomes a frozen, self-contained Python artifact: a small local decision model arbitrates every record, nothing is generated, every output is bounded, and low-confidence records abstain instead of guessing.

Agentic schema intent and the compile pipeline

STROT combines Agentic Schema Intent, Compiler & State, Code, Execution, and NLU into JSON decisions. The compile phase occurs once per intent plus schema. Inputs include intent and a sample, which synthesize into a golden gate, annotated with teacher labels, then calibrated. The result is a frozen artifact, a .py file, containing the compiled decision model. This compiled artifact captures the projection, handlers, and confidence threshold from that single synthesis step.

Runtime: every record, forever

At runtime, each record flows through a pipeline: projection, the decision model (144M parameter model running on CPU, approximately 60 milliseconds), a confidence gate, and output as typed JSON. The decision model chooses or abstains. No LLM calls occur during runtime. The marginal cost is zero. LLM calls per record are zero. This design means production serves millions of records without additional LLM expenses.

Benchmarks: compiled exceeds naive accuracy

Real models, real datasets, clean Docker containers show consistent gains from compilation. On grounded routing accuracy with 398 records, compiled achieves 92.5% versus 74.8% naive, worth plus 18 percentage points on the same model. On semantic text (AG News, unseen labels), compiled reaches 86.7% versus 72.7% naive, a +14 point improvement. Typed decisions compare near-identically to the leading hosted decision API's own example, with sentiment scoring 1.79 versus 1.8. Latency remains around 60 milliseconds, marginal cost remains $0, and LLM calls per record remain zero.

Key design guarantees

Several guarantees distinguish STROT from other LLM wrappers. The compiler scores labelings and can train a distilled head, keeping only what beats the baseline on held-out data. Output carries 0% structural hallucination because it is selection and computation over the registry, nothing generated at runtime. The model is independent, a slot for NLU: local Julia-1 by default, hosted APIs, or any drop-in specialist. Content-addressed artifact caching means schema drift triggers exactly one recompile. Golden-record promotion requires artifacts to pass known answers before they serve. Fitted guarantees include temperature scaling plus split conformal prediction, baked in: dispatched decisions are wrong at most α of all records marginally, and the rest abstain.

Installation and production workflow

STROT is available through PyPI with optional feature flags. The core package installs as pure stdlib. Add the local decision model with strot-ai[local], which includes torch and transformers. Choose a hosted decision runtime with strot-ai[hosted] for openai, or an anthropic synthesizer with strot-ai[anthropic]. To bake the local model into a production image, install strot-ai[local] and run prefetch_nlu pointing at a local directory, then set the STROT_NLU_CHECKPOINT environment variable. Never pip install julia, an unrelated package. Developers can find the full architecture, compile lifecycle, calibration semantics and caveats, deployment procedures, model swapping instructions, and API reference in DEVELOPERS.md.

MIT license and model provenance

The framework is MIT licensed. The default decision model Apache-2.0 (SupersonicLabs/Julia-1) implements ideas from the arXiv paper 2505.01636. The project provides website, documentation, benchmarks, and PyPI packages for immediate use.