LLMxRay puts local LLMs under observability

You run a local LLM and chat with it but have little insight into what actually happened. LLMxRay makes the inner workings visible without sending anything to the cloud. The interface sits between you and Ollama and reveals token-level detail, quality signals, performance numbers, and cost estimates for every interaction.

Chat diagnostics with confidence coloring

Tokens arrive one by one with coloring based on generation speed. You can watch where the model hesitates and which tokens it was unsure about. The system automatically flags problems: repetition detected through 4-gram analysis, refusal patterns including the common "as an AI language model" phrasing, gibberish marked by high non-ASCII ratio, empty responses under ten words, and truncation when the token limit is hit without finishing. Badges appear only when something is wrong.

Cache lab and decision models

A new feature in version 0.6.0 finds whether your system prompt is being recomputed from scratch each turn — often the answer appears within thirty seconds. The same release adds decision models — nimble, tev1, clef, and clef-flash — that answer with choices, probabilities and scores instead of free-form text. Give the lab a state such as a ticket or message and typed questions: a choice among named candidates, a yes/no probability, or a score on an ordered scale. Every answer comes back as the model's own probability distribution drawn as bars with the chosen candidate highlighted.

Side-by-side comparison and surgical benchmarking

Run the same prompt against two models or two temperatures and see results next to each other. Stop guessing which setting was better. The surgical benchmark scores models on real token logprobs rather than on whether the answer happened to look right. Latency percentiles P50/P95/P99 track duration and time-to-first-token. An error intelligence classifier with a timeline groups problems into seven categories.

Protocol observability and RAG knowledge bases

Fire the same prompt through Ollama's three serving protocols — native /api/chat, OpenAI-compatible /v1/chat/completions, and Anthropic-compatible /v1/messages — in parallel. Side-by-side streaming shows per-protocol metrics and an envelope-diff tab reveals how each protocol frames finish reasons, token counts, and error envelopes. Drop PDFs and documents into a knowledge base; the system chunks them, creates embeddings, and shows what RAG actually retrieves before the model ever sees the text. Embed text, visualize vectors, measure cosine similarity, and test what Matryoshka truncation costs in similarity.

Tool builder and code completion

A drag-and-drop node canvas lets you build tool definitions with bidirectional code sync — edit nodes or TypeScript and both update. Probe APIs, auto-generate schemas, and test with live execution. Code completion works for Qwen-Coder, CodeLlama, Codestral, DeepSeek-Coder, and StarCoder using two textareas (prefix/suffix) and Ollama's suffix field on /api/generate.

KV cache lab and per-user profiling

A local model reuses its KV cache only while the prompt still matches from the very first token. A single timestamp near the top forfeits everything below it. The lab finds values that change between turns, shows the exact point where reuse dies, and measures what moving them to the end actually saves. On a real 324-token prompt: four tokens reused and 64.6 ms of prefill with the timestamp at the front, 290 reused and 18.6 ms with it at the back — 3.5x faster, same words. Requires Ollama 0.33.3+. Token usage per model per day includes estimated cloud-equivalent pricing so you see what you save by running locally.

Who benefits and what runs it

Developers can debug prompts, profile latency, compare models, inspect tool calls, and track costs. Researchers run controlled experiments with consistent settings across models and temperatures. Students and educators explore model behavior visually through a built-in Educators Kit with nine interactive modules. AI team leads understand quality trends, error patterns, and resource usage across a local fleet. The interface translates into English, French, Serbian (Latin and Cyrillic), Chinese, and Arabic with RTL layout support. Community scaffolds exist for Hebrew and Japanese. The project is tested and verified against Ollama 0.40.x (verified on 0.40.3, October 2026).

  • Framework: Vue 3.5 + Composition API
  • Language: TypeScript 5.9 (strict)
  • Build: Vite 7.3
  • Styling: Tailwind CSS 4.2
  • State: Pinia 3 (store-per-concern)
  • Charts: Chart.js 4, D3.js 7
  • Canvas: Vue Flow (visual node editor)
  • Code editor: CodeMirror 6
  • Storage: IndexedDB (browser-native)
  • LLM backend: Ollama (local)

Getting started

One command starts the interface: npx llmxray. With Docker: docker run -p 5174:5174 djovaneli/llmxray. Open http://localhost:5174 and start chatting. Prerequisite: Ollama running locally with at least one model pulled, for example ollama pull llama3.2. Compatible with Ollama 0.20 and newer; recommended 0.33.3+ for prompt-cache reuse reporting. Source and further setup details live in the repository.