Reading 10,000 Privacy Policies at Once: What the Legalese Actually Reveals
Privacy policies are the primary mechanism by which organizations disclose how they collect, process, and share personal data. They are also, by design or neglect, nearly impossible for the average user to interpret. A team of researchers at the University of Michigan and the University of North Florida has built a system that reads them all—10,000 of them—and converts each one into a structured representation that can be quantitatively compared, scored, and benchmarked. The result is not just a larger dataset of annotated privacy policies but the first repeatable framework for measuring how complete, transparent, protective, and business-oriented a corporate privacy policy actually is.
The Problem with Reading the Fine Print
The challenge is not merely that privacy policies are long. They are inconsistent across organizations, vary in what they disclose, and use dense legal language that obscures rather than clarifies. Users suffer from consent fatigue; they click "accept" without reading. Researchers who attempt to compare policies face a labor-intensive bottleneck: converting unstructured natural language into structured formats requires manual annotation that is slow, expensive, and difficult to scale.
Prior work has made progress. The landmark study by Wilson et al. (2016) characterized user data collection using 15 categories. Subsequent efforts expanded this to 34 categories and 125 descriptors. LLM-based approaches have further automated the annotation process, with some systems achieving competitive performance without task-specific training. But these efforts stopped short of producing quantitative metrics that could characterize the overall quality and orientation of a privacy policy. They extracted data but did not score what the data meant.
The gap this new study fills is the distance between extraction and evaluation. The researchers ask: beyond regulatory compliance, how do we measure what makes a privacy policy good or bad, protective or exploitative? And can we answer that question across thousands of policies simultaneously?
The Pipeline: From URL to Structured JSON
The system operates as an end-to-end pipeline. Starting from an Internet domain, it retrieves the website's privacy policy, segments it into topical sections, and then extracts two kinds of structured information: the data elements the organization collects, and the practices governing those data elements.
The retrieval mechanism uses custom web crawling rather than relying on existing datasets, which allows the framework to capture up-to-date snapshots of policies that can change over time. The crawler follows privacy-related links from the homepage, discovers policies hosted behind privacy hubs, and converts HTML to Markdown. Non-English pages and documents shorter than 5,000 characters are discarded; the latter threshold filters out trivial policies from organizations with minimal data collection.
Once a policy is retrieved, the pipeline uses an LLM (OpenAI's gpt-5.1 with low reasoning effort) to segment it into logical sections labeled by topic: data elements, collection methods, purposes, handling (retention and protection), sharing, rights, cookies and tracking. Unlike approaches that rely on HTML headings alone, this method provides the LLM with the full page content, capturing secondary topics that may appear within a section without being reflected in its heading.
A Taxonomy of 237 Descriptors
The core of the system is its taxonomy. Data elements are classified into 33 categories grouped under 7 meta-categories: identity and background (58 descriptors covering personal identifiers, contact info, demographic information, education and professional details, vehicle info), digital profile (39 descriptors covering online identifiers, account and device information), physical and health profile (26 descriptors covering medical, biometric, and fitness data), financial and legal profile (38 descriptors), physical behavior (24 descriptors covering location, movement, and sensory data), digital behavior (50 descriptors covering product usage, internet activity, content generation, transactions, communications, and preferences), and other.
This represents a substantial expansion from prior work: the descriptor vocabulary grew from 125 to 237, capturing finer-grained specifications of what organizations actually collect. For each extracted data element, the pipeline infers five attributes: the category, the specific descriptor, the subjects from whom data is collected, the entities that collect it, and the collection method. A blind validation against human annotations achieved 99.0% validity for data elements and 98.8% for data practices across 500 validated samples.
Crucially, the pipeline does not simply extract data elements and practices independently. It links them semantically. When a policy states that "we retain anonymized data indefinitely," the system connects that retention practice to the specific data elements it governs and flags it as applying only to non-identifiable data. When a policy says "we share location data with advertising partners," the system records which data descriptors are shared, with whom, and for what purpose. These relational links are what enable the quantitative analysis that follows.
The Corpus: A Representative Snapshot of the Web
The evaluation corpus combines three sources. From the Russell 3000 index, the researchers identified 2,926 unique company domains, of which 78% returned qualifying privacy policy pages. From the Tranco list of top websites, they sampled 3,000 high-ranking domains and an additional 4,724 randomly selected domains, yielding qualifying text from 12.5% of the full Tranco snapshot. Across these three strategies, 10,000 domains with qualifying privacy policies were obtained.
The corpus is diverse: large public companies as well as high-traffic but non-public and smaller entities. Each company was assigned an industry sector by prompting the LLM to infer it from the homepage, achieving 92.9% accuracy against Russell 3000 ground-truth labels across 11 GICS sectors plus "Public Administration" and "Other." Of the 10,000 domains, 96.6% contained text for at least one core topic, with an average of 103.5 structured extractions per domain.
Four Metrics for Four Dimensions
Building on the structured representations, the researchers define four quantitative metrics that aggregate fine-grained extractions into high-level assessments:
- Completeness measures whether the policy covers all the data elements and practices an organization actually engages in. It has three sub-components: practice coverage (does the policy address all relevant categories?), descriptor specificity (does it specify which precise data types are collected?), and method specificity (does it state how data is collected?).
- Transparency measures how clearly the policy communicates its practices. It has two sub-components: scope (does it specify which data, subjects, and contexts the policy applies to?) and clarity (is the language unambiguous?).
- Commitment to User Protection aggregates practices related to data retention limits, security safeguards, user rights, and choices. Higher scores indicate stronger protective commitments.
- Emphasis on Business-Driven Data Practices captures the extent to which the policy describes data collection and sharing for monetization, advertising, analytics, and other business purposes.
These metrics are computed from the structured JSON representations, making them repeatable and comparable across organizations and industry sectors. The researchers demonstrate their utility by analyzing differences across sectors, examining the treatment of sensitive data, exploring the relationship between completeness and transparency, and quantifying the tension between user protection and business interests.
What the Numbers Reveal
The analysis across industry sectors shows systematic variation. Financial and healthcare organizations tend to score higher on user protection metrics but also higher on business-interest metrics, reflecting the dual pressures of regulation and data-driven business models. Social media platforms exhibit the highest emphasis on business-driven data practices, with extensive data sharing and advertising-related purposes dominating their policies.
The relationship between completeness and transparency is not straightforward. A policy can be comprehensive in what it discloses yet opaque in how it discloses it. The metrics separate these dimensions, revealing that some organizations provide exhaustive lists of data types but bury the details in impenetrable legal prose, while others are more concise but clearer.
The user protection versus business interest analysis is perhaps the most practically significant finding. The metrics quantify a tension that qualitative reading suggests but cannot measure at scale: the degree to which an organization's privacy policy commits to protecting user data while simultaneously describing practices that monetize that data. The case study on TikTok illustrates how the framework can track changes over time, capturing how a policy evolved in response to regulatory pressure or public scrutiny.
What This Enables That Did Not Exist Before
Before this framework, comparing privacy policies at scale required either labor-intensive manual review or shallow keyword-based heuristics that missed semantic nuance. The structured representations produced by this pipeline enable a new class of analyses: benchmarking an organization's policy against its industry peers, tracking how policies change over time, identifying sectors where user protection and business interests are most in tension, and providing regulators and researchers with quantitative evidence about the state of privacy disclosure across the web.
The framework is also extensible. The taxonomy can be updated as new data collection practices emerge. The metrics can be refined as the research community agrees on what constitutes a "good" privacy policy. The pipeline can be applied to new corpora, including policies in languages other than English or policies from specific jurisdictions with particular regulatory requirements.
Limitations and the Path Forward
The study acknowledges several constraints. The LLM extraction, while validated at high accuracy, is not perfect; 13% of descriptor extractions fall outside the controlled vocabulary, and some organizational nuances are inevitably lost. The corpus is weighted toward English-language policies from organizations with conventional websites. Policies provided only as PDFs were excluded, as were organizations without a web presence entirely. The metrics, while novel, are validated against the researchers' own judgments; independent expert validation of the scoring rubric remains a future direction.
Perhaps the most important limitation is that the metrics measure what policies say, not what organizations actually do. A comprehensive, transparent, protective privacy policy is a necessary but not sufficient condition for actual data protection. The framework illuminates the disclosure landscape; it does not, and cannot, verify compliance with the disclosed practices.
Nevertheless, the study represents a significant step toward making privacy policy analysis rigorous, scalable, and quantitative. For the first time, there exists a dataset and a scoring framework that allows researchers, regulators, and even informed users to ask not just "what does this policy say?" but "how does this policy compare, and how protective is it relative to the industry norm?" In a digital landscape where 57% of U.S. college students report using AI in their coursework at least weekly and data collection practices are increasingly opaque, that question matters.
Read the paper on arXiv