A Proof-of-Concept That Reads Terms of Service So You Don't Have To
Most people click "I agree" without reading the document. Terms of service, privacy policies, and cookie policies are long, full of legal jargon, and change without notice. Termsinator is a proof-of-concept project that tries to automate this reading using LLMs. It is a public registry where anyone can submit a URL and get back a structured analysis of what the site's legal documents actually say about how your data is used.
The project is open source, built in Go and Node, and runs a browser extension alongside a web interface. The core idea is simple: submit a URL, get a scored report with citations back to the original text. The harder part is making that report trustworthy enough to be useful, and the developers behind Termsinator have been candid about where the system still falls short.
How the Processing Pipeline Works
When someone submits a URL, Termsinator normalizes it and queues the job. A worker process then does deterministic crawling: it checks the root page, robots.txt, sitemap, footer links, and known legal paths, fetching up to eight candidate documents within byte and time limits. The text is split into immutable evidence passages. A Qwen 3.7 Flash model through OpenRouter runs two structured passes covering 44 evaluation criteria. Passage IDs expand into exact citations. Scores are recomputed, critical-flag gates are applied, and both JSON schemas are validated before publication.
The key design choice is separating crawling from model evaluation. The crawler is deterministic and bounded. The model only sees pre-fetched, normalized text, not raw HTML. This limits the attack surface for prompt injection and keeps model calls predictable. The worker runs in an isolated container with resource limits, a read-only filesystem, and no access to the database or private network.
The 44-criterion evaluation matrix covers data collection scope, sharing practices, user rights, deletion policies, dispute resolution, and more. Each criterion gets a score, a verdict, and a citation. The system then aggregates scores by category and produces an overall verdict. The architecture is designed for multi-model aggregation, where different models evaluate the same documents and disagreements remain visible rather than hidden behind a single number.
A Benchmark That Found Real Problems
The developers ran a ten-site benchmark against major providers: OpenAI, Google, Microsoft, Apple, Amazon, Meta, Netflix, Spotify, Adobe, and LG Smart TVs. The results are diagnostic, not published ratings, and the team is explicit about that. OpenAI scored highest at 89.8 with 100% document coverage. Apple, Netflix, and Spotify were capped at 49.9 ("high concern") due to problematic clauses. LG returned "insufficient evidence" because its privacy policy lives behind a JavaScript-rendered page the crawler could not reach.
The benchmark exposed several concrete problems. Qwen was too lenient: it gave OpenAI 27 of 44 criteria the maximum score, which may reflect protective EU policy text but is not reliable from a single inexpensive model. Product scope was ambiguous. Apple's discovery mixed store, iCloud, iTunes, website, assistant, and replacement-service terms into one bundle. Google was classified as email when its root covers dozens of products. Jurisdictions were mixed, with the VPS often receiving Hungarian or EU variants of policies while the model evaluated them as if they were universal.
The most revealing failure was citation validity. Before a conservative lexical gate was added, Qwen raised eight critical flags across Apple, Netflix, Spotify, and LG. Manual inspection found unsupported examples: a Netflix prohibition on AI training was interpreted as permission, and an LG equipment clause was cited for AI training. Passage IDs prove what text was selected, not that the model interpreted it correctly. This is the fundamental gap between structural citation and semantic understanding.
Safeguards Added After the Benchmark
The team added several safeguards based on the benchmark findings. Redirect aliases and duplicate final policy URLs are now collapsed. Final URL, title, and content determine document kind instead of the discovery link. Privacy text can no longer support terms-only criteria, and vice versa. Mixed regional URL contexts force "insufficient evidence" rather than averaging conflicting contracts. Critical verdict caps require conservative lexical support in the cited clause.
A focused post-safeguard validation reran five problematic sites. Amazon's privacy-only discovery now yields insufficient evidence rather than a score based partly on the wrong document class. Microsoft's mixed English/Hungarian discovery produces the same. Netflix's unsupported AI-training critical flag disappeared, and the uncapped output was 78.5 with 88.4% coverage. Spotify's document-class validation removed the unsupported content-appropriation cap. LG's privacy content became reachable on retry, but Qwen supported only 16.3% of the matrix, so the report stayed at insufficient evidence.
The Amazon manual follow-up fixed a specific discovery bug. The crawler had been stripping the meaningful nodeId parameter from Amazon's help-center URLs while keeping tracking parameters. The fix preserves allowlisted policy identifiers and prioritizes links from the submitted root. A rerun found the primary Terms and Privacy Notice, reached 91.1% coverage, and produced a raw score of 58.0, capped to 49.9 due to perpetual, irrevocable, fully sublicensable user-content rights and mandatory arbitration clauses.
Why the Production Profile Is Still Disabled
The processing profile remains disabled while benchmark findings are addressed. The developers list seven gates that must pass before publication: resolving one product, locale, and current policy revision per bundle; adding isolated rendering for JavaScript-only legal pages; rejecting criteria evaluated against the wrong document class; running a second independent model for critical flags; building a manually reviewed gold set; recording actual token costs; and publishing only reports that pass structural, semantic, scope, locale, and freshness gates.
At current model rates, the five-site validation mean was about 86,000 input tokens and 4,600 output tokens per site. At illustrative rates of $0.03 per million input and $0.13 per million output, that works out to roughly $3,850 per million evaluations after OpenRouter fees, retries, and overhead. The cost is not prohibitive at small scale, but it requires recording actual billing rather than inferring from token counts.
The architecture is designed for multi-model aggregation, but the current benchmark uses only Qwen. The developers acknowledge that a single inexpensive model is too lenient and inconsistent to establish a public ranking. The aggregation design shows every model evaluation and derives criterion consensus with medians. Disagreements and minority critical flags remain visible rather than hidden.
The Browser Extension and the Public Interface
The Manifest V3 extension works in two modes. Click-only mode requests the active tab and contacts the API when the popup opens. Automatic mode sends only hostnames on navigation events. Neither mode reads page content or sends paths, queries, titles, or URLs. The extension displays report age, aggregate verdict, coverage, model count, disagreement status, top risks, and a link to the full comparison.
The web interface is a minimal static Astro build with no analytics or third-party scripts. It provides anonymous URL submission, a progress view, a category-filtered rankings list, and methodology documentation. The API exposes structured endpoints for submissions, status checks, summaries, reports, and category queries. All public endpoints are read-only. Workers claim jobs through a private Docker network and write through narrowly scoped database procedures.
The project is at proof-of-concept stage with 24 commits. It requires Go 1.19+, Node 22, pnpm 10, and PostgreSQL 16. The production deployment uses blue/green releases with Caddy as the edge, immutable image tags, and atomic upstream switching. The developers note the shared VPS has only about 1.8 GiB available, so conservative limits apply. The live demo is at a sslip.io address pending infrastructure decisions about TLS and domain configuration. A license must be selected before the repository is fully published.