IBM's Granite team released LogitScope, an open-source Python framework that analyzes large language models by computing information-theoretic metrics directly from token probability distributions. Instead of interpreting what a model says, LogitScope measures how the model distributes probability mass across its vocabulary at each position, turning opaque outputs into quantifiable signals about uncertainty, confidence, and decision-making.
Measuring What the Model Does, Not What It Says
At each token position, a language model produces a probability distribution over its entire vocabulary. LogitScope extracts six metrics from these distributions, each capturing a different property. Surprisal (negative log probability of the chosen token) measures how unexpected the output was. Entropy measures how spread out the probabilities are, a proxy for overall uncertainty. Varentropy, the variance of surprisal values, detects when a model is torn between multiple distinct options. Skewentropy captures whether probability mass is concentrated or dispersed asymmetrically. Perplexity, the exponential of average surprisal, provides a cumulative quality score across a sequence. Direct probability gives a simple confidence measure from 0 to 1.
These metrics are computed lazily and cached, so users pay the cost only for the metrics they actually access. The framework handles tokenization, model inference, softmax computation, and metric calculation in a single pipeline, working with any HuggingFace causal language model out of the box.
Use Cases That Do Not Require Understanding the Text
The metrics enable several practical applications without semantic interpretation. Regions with high entropy and varentropy often indicate hallucinated content, where the model generates facts with high confidence but shows uncertainty in surrounding tokens. Tracking perplexity and average entropy over time can detect model degradation in production. Comparing multiple model responses to the same prompt becomes a quantitative exercise: lower perplexity and higher token probabilities indicate more natural outputs.
For fine-tuning, LogitScope measures before and after metrics on validation sets. Improved models show lower perplexity and higher probability on expected outputs without increased entropy on known-answer questions. Prompt engineering benefits from observing how different phrasings affect model confidence: good prompts reduce entropy while maintaining high probability on correct tokens. The framework also supports ablation studies, uncertainty quantification, and cross-linguistic or cross-domain distribution analysis.
Programmatic and Interactive Access
The core API loads a model and tokenizer, initializes a LogitScope instance, and returns a Results object from a measure call. Results expose metric properties as lists and support iteration over tokens with selected metrics, plus top-k inspection at any position showing the alternative tokens the model considered. The Results object also provides access to raw logits, probability distributions, log probabilities, and decoded token text.
The optional web UI, built on FastAPI and Uvicorn, adds visual exploration. It displays color-coded token visualization by metric, live statistics with running averages, interactive token explorer with top-k alternatives, distribution plots, metric toggles, and real-time WebSocket streaming. The UI launches with a single command and accepts flags for model selection, device, and port.
Compatibility and Requirements
LogitScope runs on Python 3.11 or later and requires PyTorch 2.7+, Transformers 4.51+, and NumPy 1.17+. It supports CPU, CUDA, and Apple Silicon MPS devices. The UI adds FastAPI 0.115+ and Uvicorn 0.34+ as optional dependencies. Development tooling uses Ruff for formatting and linting.
The installation uses uv for environment management. The package installs in editable mode, with optional extras for the UI and development. The codebase follows a plugin architecture for custom metrics, allowing teams to extend the framework with domain-specific measurements beyond the built-in set.
Farhan Ahmed, Yuya Jeremy Ong, and Chad DeLuca published the accompanying paper on arXiv in 2026, describing the framework's approach to analyzing LLM uncertainty through information metrics. The tool is part of IBM's Granite debug-tools repository on GitHub.
For teams deploying LLMs in production or evaluating model behavior during development, LogitScope offers a way to move beyond subjective assessment of outputs. The probability distributions are already computed during inference; LogitScope just makes them visible and measurable. The result is a debugging tool that works regardless of the language, domain, or specific model being used.