Most teams evaluate their AI agents on a tiny fraction of production traffic because the LLM judge is both slow and expensive. OpenLayer AI released jevals, an open-source Python framework that packs every quality, safety, and security check for an agent trace into a single HTTP request for a few thousandths of a cent in a few hundred milliseconds.

Why LLM-as-judge hits a wall for agents

Tools like Ragas gave the field its standard metrics: faithfulness, answer relevancy, context precision. But each metric requires its own LLM call, often multiple. Running four Ragas metrics on one sample takes six to eleven round trips. That costs enough that most teams sample 1% of traffic and run nightly, never anywhere near the live request path.

Agents compound the problem. Traces are longer, there are more things to verify (tool selection, tool result usage, scope adherence, whether tool results contain injection attempts), and the judge itself is non-deterministic. LangChain published a comparison on identical traces showing GPT and Claude judges exhibited 92x to 913x the score variance of Jev, the decision model jevals is built on. A judge that produces different scores on the same input makes a poor regression test.

How jevals changes the math

Jev does not generate text. It takes a state dictionary and a set of typed questions, each one yes/no, a choice from labeled options, or a score on a rubric. It returns a calibrated probability for every question in a single forward pass. Questions evaluate independently and in parallel, so asking 40 questions costs roughly the same latency as asking one.

Through Vercel AI Gateway, the project measured p50 244ms and p95 371ms per request. Pricing is $0.042 per million input tokens with no output tokens billed. For the quickstart example (agent trace with 9 checks), the actual cost was $0.00006 in 0.50 seconds.

Within a week of Jev's launch, two open-weight models appeared speaking the same API: Kev (Qwen3-based, runs on a 32GB Mac) and Laya (ModernBERT, approximately 10ms on Apple Silicon). The request shape, state plus a dictionary of typed questions, looks stable enough to become a standard.

What you get out of the box

The package ships 37 built-in evals across three modules:

  • jevals.agent: ToolChoice, ArgumentValidity, UsedToolResult, Grounded, StayedInScope, StepProgress, LoopDetection, GoalCompletion, PlanAdherence, Quality, ToolCallRisk, TrajectoryMatch, ToolCallF1
  • jevals.security: PromptInjection, IndirectInjection, Jailbreak, GoalHijacking, SystemPromptLeakage, ExcessiveAgency, PII, PHI, SecretsExposure, Toxicity, Bias, NonAdvice, TopicAdherence
  • jevals.quality: Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecall, Hallucination, Correctness, Completeness, Coherence, InstructionFollowing, Refusal, CustomRubric

PII and PHI detection uses a two-step pipeline. Presidio (or a regex-and-checksum fallback with extra recognizers for Brazilian CPF, US NPI, medical record numbers, and health plan IDs) finds entities first. Then one question to the model determines whether the entity represents health information about an identifiable person, a support email address, or something else. Entity detection alone cannot make that distinction, which is the primary source of false positives in PII detection.

Every eval is a class with three methods: state() extracts what the model should look at from the trace, questions() returns typed questions about that state, and reduce() turns the answer probabilities into a score. When you pass multiple evals to evaluate(), their states merge and all questions go out in one request.

Running in the agent loop as guardrails

Because the same evals are fast and cheap enough, they can run inside the request path, not just offline. A gate pairs an eval with a policy that maps answers to allow, escalate, or block. You can define gates in Python or YAML.

For a support agent with lookup_order, issue_refund, send_email, and run_sql tools, the gate configuration might look like this: a tool_call_risk gate scores every proposed call against whether it is destructive, grounded in the customer's request, and reversible. lookup_order passes (approve=1.00, grounded=0.95). run_sql("DELETE FROM orders") escalates (destructive=0.96, grounded=0.21). issue_refund for $500 that nobody asked for also escalates, even though a smaller requested refund would pass on the other criteria, because the policy sends anything that moves money to a human regardless of grounding score.

On the ingress side, an IndirectInjection gate scans tool results before they reach the model. PHI(action="redact") modifies the tool result to strip protected health information before it enters the context. LoopDetection gates escalate when an agent revisits the same steps within a window.

The integrations are straightforward. For OpenAI Agents SDK, there are input_guardrail, output_guardrail, and guard_tools wrappers. LangGraph gets a node before the tool node. Claude Agent SDK uses a PreToolUse hook returning allow, ask, or deny. For custom loops, a single Gate.check() call works.

Benchmark numbers against Ragas

The project includes a jevals bench --ragas command that runs the four Ragas-equivalent metrics (faithfulness, answer relevancy, context precision, context recall) through both libraries on the same 20-row RAG dataset. The comparison measured on 2026-09-20:

  • Ragas with gpt-4.1-mini: 6.0 LLM calls plus embeddings per sample, 4,390 input tokens, $2.60 per 1,000 samples, 22 to 35 seconds for 20 rows.
  • jevals with gpt-4.1-mini emulating Jev: 1 request, 736 input tokens, $0.46 per 1,000 samples, 4 seconds.
  • jevals with Jev through Vercel: 1 request, 824 input tokens, $0.03 per 1,000 samples, 0.8 seconds.

All three rows agreed on verdicts (faithfulness 0.90 to 0.92, context precision and recall 1.0). Real Ragas on the OpenAI API would be closer to 8 requests per sample since it asks for three completions where the OpenRouter row only got one.

Adding six security evals on the jevals side means adding more questions to the same request. On the Ragas side it would mean six more separate LLM calls. JevBench, an independent benchmark, puts Jev at 83 to 87% accuracy on Banking77 and CLINC150 classification tasks, comparable to the smallest LLMs, though calibration varies by task. LangChain's agent eval had Jev agreeing with a human on pass/fail 100% of the time across 500 repetitions on five traces, compared to 80% for Claude.

Practical workflow for teams

The CLI runs evals over a dataset of traces: jevals run traces.jsonl --evals agent.tool_choice,agent.grounded,security.indirect_injection produces pass rates and mean scores per eval, writes per-trace results to JSONL, and shows the worst failures for manual inspection.

jevals calibrate takes labeled data and fits the decision threshold against your labels, reporting the error rate at each threshold. Each row shows the tradeoff between wrong approvals and missed escalations, with Brier, ECE, and AUROC metrics.

Backend resolution is automatic from environment variables. AI_GATEWAY_API_KEY picks Jev through Vercel (the easiest path today). TYPESAFE_API_KEY goes direct to TypeSafe (waitlist access). KEV_BASE_URL runs Kev self-hosted. JEVALS_BACKEND=laya uses Laya in-process on Apple Silicon. OPENROUTER_API_KEY emulates Jev through any chat LLM, which is slower and costs more but works immediately. You can also pass a backend string explicitly to evaluate().

The MCP server lets Cursor, Claude Code, and Copilot look up, write, validate, and run evals without guessing at the API. A TypeScript package is next; eval definitions are already JSON and should drop into the AI SDK's experimental_evaluate without much work.

What it does not do

It does not generate test sets, it has no dashboard, and it will not replace an LLM judge for work requiring multi-step reasoning or written critiques. The underlying models are a week old. The project recommends calibrating on your own data and keeping a human on irreversible actions. The framework adapters (OpenAI Agents SDK, LangGraph, Claude Agent SDK) are written to SDK docs and tested against fakes, not run live yet. Alpha quality, MIT licensed, Python 3.10+.