Nine Numbers That Spot the Ghostwriter: How Stylometry Catches GPT-Assisted Essays

Academic integrity offices are drowning in a new kind of plagiarism: not copied text, but co-authored text. When a student pastes a prompt into ChatGPT, paraphrases the output, and submits it as their own, traditional similarity checkers see nothing unusual. The words are original; the authorship is not. A study by Rajesh Kumar, Nabeel Siddiqui, and Alexander Fuchsberger demonstrates that the difference shows up in nine measurable stylometric features, and that a Random Forest classifier trained on those features can separate GPT-assisted from independently authored student writing with an ROC-AUC of 0.87 on unseen participants.

Why the Writing Process Matters More Than the Output

Most AI detectors—DetectGPT, watermarking schemes, adversarial classifiers—operate on assumptions about how models generate tokens. They require access to probability distributions, generation logs, or proprietary APIs. None of that is available to an instructor staring at a submitted PDF. Keystroke-dynamics approaches capture the writing process but need instrumented editors, which rules out retrospective analysis of existing assignments.

The authors take a different tack: they ask whether the final text itself carries enough stylistic signal to reveal its provenance. Stylometry, the statistical study of writing style, has successfully attributed authorship for decades. The hypothesis is that even when a student paraphrases GPT output, the paraphrased text retains lexical and grammatical fingerprints of the model's tendencies. The study tests this hypothesis under controlled conditions: 90 participants, two writing sessions per participant, one independent and one with ChatGPT assistance followed by mandatory paraphrasing.

The Dataset: Paraphrased, Not Copied

The IIITD-BU (Paraphrased) dataset, originally collected by Mehta et al. (2026), comes from students in a Large Language Models course at IIIT Delhi. Each participant answered two prompts—one explaining how LLMs work, the other analyzing their strengths and weaknesses in education—first without assistance, then with ChatGPT but with copy-paste disabled and an explicit instruction to paraphrase before typing. This design isolates the effect of GPT-assisted composition from verbatim copying.

Independently authored responses averaged 620 words; GPT-assisted responses averaged 515 words. To prevent document length from becoming a trivial classifier feature, the researchers divided every document into overlapping 250-word windows with a 125-word stride. Each window received its own feature vector. Windows from the same participant were never split across train and test sets: 72 participants (144 documents) formed the training set, and 18 held-out participants (36 documents) formed the test set. This disjoint-user design is critical—it ensures the classifier is evaluated on genuinely unseen writers, not just unseen text from known writers.

Nine Features, Four Linguistic Categories

Each window was represented by nine interpretable features:

  • Lexical diversity: Type-Token Ratio (unique tokens divided by total tokens), Hapax Ratio (tokens occurring exactly once), and Word Entropy (Shannon entropy of the token frequency distribution).
  • Lexical density: Non-Stopword Ratio (proportion of tokens not on the 175-word Snowball English stop list).
  • Grammatical composition: Noun Ratio, Verb Ratio, Adjective Ratio, and Adverb Ratio, each computed over UD-pipe POS tags.
  • Sentence structure: Sentence-Length Variability, the standard deviation of sentence lengths in tokens.

All features were centered and scaled using training-set statistics only. The final document-level prediction was the median of window-level probabilities, a choice that dampens outliers while preserving probabilistic information that hard majority voting would discard.

What the Numbers Say About Human vs. AI Style

Paired t-tests across the 90 participants reveal a consistent stylistic signature. GPT-assisted windows showed significantly higher values on every lexical feature: TTR (0.65 vs. 0.60), Hapax Ratio (0.51 vs. 0.44), Word Entropy (6.94 vs. 6.74), Non-Stopword Ratio (0.64 vs. 0.59), Noun Ratio (0.27 vs. 0.24), Verb Ratio (0.14 vs. 0.13), and Adjective Ratio (0.08 vs. 0.08, barely significant). Independently authored windows were higher on Adverb Ratio (0.05 vs. 0.04) and Sentence-Length Variability (14.67 vs. 13.07).

The effect sizes are meaningful. Non-Stopword Ratio and Noun Ratio have point-biserial correlations of 0.50 and 0.47 with the GPT-assisted condition. Adverb Ratio correlates at -0.34 with GPT assistance. These are not subtle differences; they are detectable with simple statistics on 250-word segments.

A control experiment confirms that document length alone is not the driver. A classifier using only total word count reached only 0.68 ROC-AUC. Feature-group ablations showed that lexical features carry most of the signal, part-of-speech features add incremental value, and Sentence-Length Variability alone performs near chance. Replacing TTR and Hapax Ratio with moving-average versions preserved performance, indicating the signal is robust to the specific diversity metric.

Eight Classifiers, One Clear Winner

The researchers evaluated Logistic Regression, LDA, QDA, Naive Bayes, Random Forest, linear SVM, k-NN, and a multilayer perceptron. Hyperparameters were tuned via Bayesian optimization on the training partition only. Random Forest ranked first in both repeated 10-fold cross-validation (mean validation ROC-AUC 0.838) and a nested 5x5 validation check. Its final configuration: 500 trees, max three candidate features per split, minimum node size 10.

On the held-out test set of 18 unseen participants, Random Forest achieved:

  • ROC-AUC: 0.87 (95% CI: 0.75–0.98)
  • F1-score: 0.84 (95% CI: 0.74–0.94)
  • False Positive Rate: 22.2% (4 of 18 human documents flagged as AI)
  • False Negative Rate: 11.1% (2 of 18 AI documents missed)

The confidence intervals are wide because the test set is small—18 documents per class. A sensitivity analysis across five train-test ratios (60/40 through 80/20) showed mean FPR ranging from 12.4% to 15.8% and mean FNR from 23.7% to 24.8%, with considerable variance across splits. The authors correctly refuse to present this as a deployable detector; it is a proof of concept that the signal exists and is interpretable.

Interpretability That an Instructor Can Actually Read

SHAP analysis on the Random Forest model reveals which features push a prediction toward "GPT-assisted." Globally, Hapax Ratio is the most influential, followed by Non-Stopword Ratio, Noun Ratio, Adverb Ratio, and Type-Token Ratio. Verb Ratio, Sentence-Length Variability, and Word Entropy contribute less; Adjective Ratio contributes minimally.

At the window level, the SHAP beeswarm plots show that the direction of each feature's contribution varies with its value, reflecting the non-linear decision boundaries of the tree ensemble. For a correctly classified GPT-assisted window (predicted probability 0.947), Non-Stopword Ratio and Hapax Ratio made the largest positive contributions. For a correctly classified human window (predicted probability 0.077), Hapax Ratio, Noun Ratio, Non-Stopword Ratio, and TTR pushed the prediction toward "human."

This is the paper's most practical contribution: when the model flags a document, the SHAP values tell you exactly which linguistic characteristics triggered the flag. An instructor can verify: "Yes, this essay does use an unusually low proportion of adverbs and an unusually high proportion of nouns." The evidence is inspectable, not a black-box score.

Where the Method Falls Short

The limitations are substantial and honestly reported. The study covers 90 participants from one course at one institution writing on one topic. The GPT-assisted condition is narrowly defined: paraphrasing ChatGPT output with copy-paste disabled. Other assistance modes—brainstorming, outlining, iterative editing—are untested. All participants completed the independent session first, so session-order effects are confounded with condition. No demographic or language-background analysis was performed, so fairness across groups is unknown. Writing styles and model outputs both evolve; today's stylometric fingerprints may not hold next year.

Most critically, the 22.2% false positive rate on the held-out set means that in a class of 30 students, roughly seven genuinely authored papers would be incorrectly flagged. The 95% Wilson confidence interval (9.0%–45.2%) is too wide for any high-stakes decision. The authors position their method as a decision-support tool that triggers human review, not as an automated judge. That is the correct framing.

What This Means for the Next Semester

The study demonstrates that you do not need a neural network trained on millions of examples to detect AI-assisted writing. Nine hand-crafted features, computed with standard NLP tooling on 250-word windows, provide measurable separation. The sliding-window design handles variable document lengths gracefully. The disjoint-user evaluation protocol is a model for how such studies should be run—if you don't hold out users, you are measuring author recognition, not AI detection.

For practitioners, the immediate takeaway is that stylometric screening is feasible with open-source tools: quanteda for tokenization, udpipe for POS tagging, any ML library for Random Forest, SHAP for explanations. The false positive rate demands human-in-the-loop review. The interpretability means that review can be grounded in specific linguistic observations rather than opaque probabilities.

For researchers, the paper opens several doors. How little text is needed for reliable classification? How do the features behave under adversarial paraphrasing or style-transfer attacks? Can stylometric features be combined with behavioral signals when keystroke data is available? Do the same features generalize across LLMs, languages, and disciplines? The current results are a baseline, not a finish line.

The broader lesson is that AI-generated text, even after human paraphrasing, leaves statistical footprints in vocabulary richness, part-of-speech distribution, and sentence rhythm. Those footprints are visible to anyone willing to count. In an academic landscape where AI use is routine—57% of U.S. college students report weekly coursework use per the 2026 Lumina-Gallup study—having a transparent, auditable screening method matters more than having a marginally more accurate black box.

Read the paper on arXiv