Every production AI pipeline eventually confronts the same uncomfortable question: the model that scores highest on your eval suite isn't the one that actually works in the wild. It might hallucinate citations, exceed latency budgets, or return responses that fail schema validation — all while posting a deceptively clean accuracy number. A new open-source framework called reliopt, published by researcher obielin on GitHub, proposes a different approach to the problem by treating AI program optimization as a constrained multi-objective engineering task rather than a score-maximization exercise.
The Core Problem with Score-Only Optimization
Most AI agent frameworks today evaluate candidates using a single aggregate metric or a weighted combination of metrics. This approach has a fundamental flaw: collapsing accuracy, cost, latency, and robustness into one number conceals the actual trade-offs involved. A program might achieve 98% accuracy at three times the latency budget, or hit low cost by silently violating a groundedness requirement. Weighted-sum optimization rewards candidates that game the aggregate rather than those that genuinely perform well across all dimensions.
reliopt addresses this by separating behavioral constraints from optimization objectives. The framework introduces a contract system — hard and soft requirements such as groundedness thresholds, schema validity checks, and tool scope restrictions — that act as gates. Any candidate that violates a contract is excluded from consideration entirely, regardless of how well it performs on other metrics. In the framework's demo, six candidate programs were tested, and five were rejected outright for contract violations before a final Pareto-optimal candidate was selected.
How the Framework Works
reliopt wraps arbitrary Python callables and treats them as composable programs. The developer defines a pipeline, attaches contracts and objectives, and then compiles the program against a training dataset. The core loop follows a pipeline of stages: the program adapter ingests the callable, contracts filter candidates, objectives define what to optimize, a perturbation engine generates stressed variants of the training data, a Pareto compiler computes the non-dominated frontier, and an evidence card documents the final selection.
The perturbation engine is one of the more interesting pieces. Rather than evaluating candidates on nominal examples alone, reliopt automatically generates perturbed variants — reordered clauses, injected distractors, and domain-specific perturbations defined by the developer. This means a program that looks robust on clean data might crumble when faced with reordered instructions, and reliopt surfaces that weakness before deployment.
Pareto Frontiers and Profile Selection
Once contracts have filtered the candidate pool, reliopt computes the Pareto frontier — the set of non-dominated programs where improving one objective would necessarily degrade another. The developer then selects a profile that matches their operational priorities: quality_first for maximum accuracy, balanced for even trade-offs, low_cost for budget-constrained deployments, or high_reliability when uptime and consistency matter most.
The framework also provides component attribution through zero-ablation testing. For pipelines composed of multiple named steps, reliopt can swap a single step for a passthrough and measure the resulting score delta. This answers not just whether a pipeline works, but which component is driving its performance — a diagnostic capability that is often more valuable than a single aggregate score.
Current State and Limitations
reliopt is labeled as an early v0.1 release, and the author is upfront about that. The core loop works end-to-end today, but several components are intentionally thin placeholders with clearly marked seams. The framework currently wraps arbitrary Python callables, with adapters for popular agent frameworks planned for future releases. Documentation in the repo points to an ARCHITECTURE.md file distinguishing what is functional from what is stubbed, and a CHANGELOG tracking shipped features.
The demo itself runs against a fake, deterministic RAG agent shipped with the repository — meaning zero API calls and zero cost. The output, while illustrative, should not be taken as representative of real-world performance. For developers wanting to run it themselves, the project includes a runnable walkthrough at examples/rag_demo.py.
Where This Fits in the Landscape
reliopt sits at the intersection of several active research areas. The project references related work in per-output constraint mechanisms — systems that enforce rules on individual model outputs rather than filtering entire candidates. It also connects to evaluation and red-teaming frameworks, as well as multi-objective prompt optimization techniques. Unlike these approaches, reliopt operates as a meta-layer above existing agent code rather than replacing any framework outright.
For teams building agent pipelines where a single misbehaving component can cascade into production failures, the contract-first filtering and perturbation-based stress testing offer a pragmatic middle ground between "hope the eval scores are right" and "manually test every edge case." The framework is open-source under the MIT license, and the author is actively seeding issues labeled for first-time contributors.