CacheVerifier, check if verifying LLM cache hits helps your traffic

CacheVerifier provides a verification layer for semantic caches, checking gray-zone cache hits with a cross-encoder before answers are served. The hosted API verifies given query and candidate cached answer pairs, approving or rejecting them so a similarity match that is close but wrong doesn't become a silent error in your application.

CacheVerifier does not run your cache or perform similarity search. Your cache backend does its own lookup first; you call verify() only on the candidates in the similarity "gray zone", where a plain threshold match might be wrong. The Python client installs via pip and requires Python 3.9+. The only runtime dependency is httpx — optional extras add GPTCache integration, health check functionality with torch and sentence-transformers, or fine-tuning capabilities.

After installing and getting a free API key from cacheverifier.com, you can verify candidates like this:

from cacheverifier import CacheVerifier
cv = CacheVerifier(api_key="cv_...")
query = "how do I cancel my subscription"
candidate = "Go to Settings > Billing > Pause subscription for a month."  # from your cache
result = cv.verify(query, candidate)
if result.approved:
    answer = candidate                       # verified hit — skip the LLM call
else:
    answer = call_your_llm(query)            # not trustworthy — fall through

verify() returns a VerifyResult with fields: approved (serve cached answer or fall through), score/threshold (approved if score >= threshold), model_version ("stock", "v" fine-tuned, "cold_start_fail_closed"/"cold_start_auto_pending", or "verify_unavailable"), latency_ms (server-side inference time), and degraded (True when the call failed and this result was synthesized client-side).

verify() runs inline with your traffic with conservative defaults: 1s timeout separate from the 10s timeout used for fine-tuning/feedback/monitoring calls. Warm verification is tens of milliseconds server-side; 1s covers the network round trip and a cold model load after a deploy without letting a stuck verifier stall your request. On timeout, connection error, or 5xx, verify() does not raise — it logs a warning and returns a VerifyResult with degraded=True and approved=False, so you fall through to your LLM exactly as you would on a cache miss. A4xx (bad key, bad request, rate limit) still raises CacheVerifierError.

You can set fail_open=True to return approved=True on an outage — only if a stale-or-near-miss answer is acceptable for that traffic. There is no formal uptime SLA yet, which is the other reason the fallback path is a built-in default rather than left to you.

Integrating with GPTCache

Drop the verifier into a GPTCache pipeline as its similarity evaluator — no fork required:

from gptcache import cache
from cacheverifier.integrations.gptcache import CacheVerifierEvaluation
evaluator = CacheVerifierEvaluation(api_key="cv_...")
cache.init(similarity_evaluation=evaluator, ...)

When you learn a served hit's real outcome:

evaluator.report_feedback(query, answer, was_correct=False, similarity_score=0.9)

Once you have ~20+ feedback rows (the service found fine-tuning is often a net negative below ~1,000 on the hardest data), train a verifier on your own gray-zone labels:

job = cv.finetune()                       # or cv.finetune(target_risk=0.01, cost_ratio=5.0)
job = cv.get_finetune_job(job["id"])      # poll until status == "done"
print(job["auc_baseline"], job["auc_tuned"])

A model can finish as "held_for_review" — promote it explicitly:

if job.get("result_model_version"):
    cv.activate_model_version(job["result_model_version"])

cv.dry_run([...]) reports the same baseline-vs-tuned AUC on examples you pass directly, without writing anything or deploying a model. cv.dry_run() still uploads your examples to the API. If that's a blocker — a compliance review, or just not wanting production traffic to leave your network — run the identical stock-vs-fine-tuned evaluation entirely on your own machine by installing "cacheverifier[healthcheck]" and running cacheverifier healthcheck traffic.jsonl or cacheverifier healthcheck traffic.jsonl --emit-summary summary.json. traffic.jsonl is a JSON array or JSONL of {"query", "candidate_answer", "was_correct"} rows in arrival order (the train/calibrate/test split is chronological, matching the hosted service so the numbers are comparable). Optional per row: "stale": true. Nothing is sent anywhere — the base model downloads once from Hugging Face, then it's fully offline. --emit-summary writes an aggregate-only JSON file (AUCs, counts, rates — no query or answer text) that's safe to share for a human read.

Choosing the base model

The CLI supports these base models, so local and hosted numbers stay comparable:

The English bases read most Chinese characters as unknown tokens, so on Chinese traffic their result doesn't mean anything — the CLI warns when it sees mostly CJK text on an English base. You can also pass any Hugging Face model id or a local model directory. If Hugging Face is unreachable from your network, set HF_ENDPOINT to a mirror, or download the model elsewhere and pass its directory.

API endpoints

verify(query, candidate_answer)                    POST /v1/verify
verify_batch(pairs)                              POST /v1/verify/batch
feedback(...) /feedback_batch(items)               POST /v1/feedback //batch
finetune(...) /dry_run(examples, ...)              POST /v1/finetune/jobs //dry-run
get_finetune_job(id) /list_finetune_jobs()         GET /v1/finetune/jobs[/id]
activate_model_version(id)                       POST /v1/finetune/model-versions/{id}/activate
drift_status()                                     GET /v1/monitor/drift-status
gray_zone_threshold()                              GET /v1/monitor/gray-zone-threshold
usage() /savings()                                 GET /v1/usage/status //savings

Non-2xx responses raise CacheVerifierError (.status_code, .detail). The research repository is separately licensed; this client is MIT.