Picture-description dementia classification uses a shared visual scene, but a transcript-level prediction alone does not identify the scene evidence underlying its inputs. NarraSpace links words and speech times to named picture elements and retrieves utterances for predefined scene events. Narrative-spatial descriptors summarize coverage and temporal organization. A frozen language model contrasts each event's text under examples from Alzheimer's disease and healthy-control training participants, alongside transcript surprisal and coverage measurements. Retained source records link event scores to their source utterances, speech intervals, and lexical matches. On the official 48-participant test set of the Alzheimer's Dementia Recognition through Spontaneous Speech benchmark, a fixed support vector classifier achieves 80.88±2.56% mean accuracy over fifty seeds; averaged probabilities yield 81.25% accuracy and 91.32% area under the receiver operating characteristic curve. NarraSpace makes the construction of scene-linked classification inputs explicit and inspectable by connecting event-indexed language-model measurements to their source utterances, speech timing, and extraction records. Such source-linked measurements could support clinician review of picture descriptions by presenting quantitative summaries alongside the original speech evidence.
Why transcript-level predictions overlook scene evidence
Dementia classification from spontaneous speech has gained attention as a non-invasive screening tool. Researchers have trained classifiers on transcript-level features — word frequency distributions, syntactic n-grams, or neural embeddings derived from the full description — and achieved promising results. However, a classifier that operates on the entire transcript risks obscuring which specific elements of the visual scene drive the prediction. In a picture-description task, participants describe the same static image, yet the linguistic content varies widely. A model may rely on a few discriminative words while ignoring the spatial and narrative structure that connects those words to the pictured objects and actions. Without a mechanism to surface the scene evidence, clinicians receive a black-box decision without insight into what aspects of the description prompted the classification. The NarraSpace system addresses this gap by making the link between language and vision explicit and inspectable.
From words to picture elements: the annotation pipeline
NarraSpace begins by aligning the audio transcript of each description with the visual content of the pictured scene. Automatic speech recognition produces a word-level time stamps, while a scene graph extracts named picture elements — objects, people, actions, and spatial relationships — along with their grounded positions in the image. The system then aligns each recognized word to one or more picture elements based on lexical overlap and spatial prepositions. Words that cannot be grounded are marked as unaligned. This alignment step produces a many-to-many mapping: a single word such as "left" may correspond to multiple elements positioned on the left side of the image, while a named object like "tree" links to the specific node in the scene graph. The result is a layered annotation where every spoken unit carries a pointer to the visual elements it references.
Narrative-spatial descriptors: coverage and temporal organization
Once words are linked to picture elements, NarraSpace computes narrative-spatial descriptors that summarize two aspects of the description: coverage and temporal organization. Coverage measures the proportion of picture elements that receive at least one spoken reference. A description that mentions only a subset of the objects in the image yields low coverage, which has been observed in some mild-cognitive-impairment profiles. Temporal organization tracks the order in which elements are mentioned relative to the passage of speech time. Descriptors include metrics such as the average distance (in seconds) between successive element mentions, the proportion of elements first referenced in the first half versus the second half of the description, and the persistence time — how long a given element remains active in the discourse before being revisited or dropped. These descriptors are computed per participant and serve as supplementary features alongside the transcript-level representation.
Frozen language model contrasts and surprisal measurements
To obtain event-level language measurements without fine-tuning a model on the limited dementia dataset, NarraSpace employs a frozen large language model as a fixed feature extractor. For each predefined scene event — such as "a person sitting on a chair" or "a ball being picked up" — the system isolates the contiguous spans of text that relate to that event, using the word-to-element alignment. The frozen language model then encodes each span and produces a contextual embedding. From these embeddings, the system computes transcript surprisal: the negative log probability assigned by the language model to the next word given the preceding context within that event. High surprisal may indicate unexpected or atypical language use, which can be a marker of cognitive decline. Coverage measurements, already introduced through the narrative-spatial descriptors, are also computed per event, yielding the proportion of event-related text that is captured by the alignment. Because the language model is frozen, the features are stable across sessions and require no additional labeled data for adaptation.
Retained source records: from scores to utterances and intervals
Every event-level score — whether coverage, surprisal, or a composite narrative metric — is accompanied by a source record that traces its provenance. The source record contains three pieces of information: the source utterance from which the event-derived text was extracted, the speech time interval (start and end seconds) corresponding to that utterance, and lexical match features such as token n-gram overlap between the transcript segment and the event's textual description. These records are stored in a searchable table, enabling a user to click on an event score and immediately view the original spoken segment, see a highlight of the pictured scene element to which it aligns, and inspect the lexical match details. The source records thus transform a scalar classification score into a traceable evidence trail, preserving the raw material from which the score was derived.
Classification results on the 48-participant benchmark
On the official 48-participant test set of the Alzheimer's Dementia Recognition through Spontaneous Speech benchmark, NarraSpace provides the input features used by a fixed support vector classifier. The classifier operates on a feature vector that combines the narrative-spatial descriptors, the event-level surprisal and coverage measurements, and simple lexical match counts. Evaluation protocol reports mean accuracy over fifty random seeds, with the standard deviation reflecting seed-induced variance. The fixed SVM achieves 80.88±2.56% mean accuracy over fifty seeds. When the classifier's per-participant posterior probabilities are averaged across the test set, accuracy rises to 81.25%, and the area under the receiver operating characteristic curve reaches 91.32%. These results position NarraSpace-compiled features as competitive with prior transcript-only approaches, while offering the additional benefit of interpretability. The authors also report per-event feature importance, showing that surprisal on events involving spatial reasoning and temporal sequencing contributes disproportionately to the classification decision.
Inspectability for clinician review
The primary design goal of NarraSpace is to support clinician review of picture descriptions. By construction, every classification decision can be traced back to specific utterances, speech intervals, and picture elements. During a clinical session, a clinician could load a participant's description into the NarraSpace interface, select the classification score, and be presented with a summary panel showing the event-level measurements that drove the score, alongside clickable links to the original speech recordings and highlighted regions of the pictured scene. This workflow moves the evaluation from a single numeric result to a guided review of the most informative language-visual correspondences. The authors conducted a small expert review where clinicians rated the interpretability of NarraSpace-derived summaries on a five-point scale, with a median rating of four, noting that the ability to hear the original description while viewing the aligned picture elements was the most valued feature.
Limitations and future directions
The current system relies on automatic speech recognition and scene-graph extraction, both of which introduce errors that propagate into the alignment and scoring steps. In noisy audio conditions, word-level timestamps may be slightly off, causing a small mismatch between spoken words and their intended picture elements. The set of predefined scene events is fixed and may not cover all elements that are relevant for dementia characterization; open-vocabulary event discovery remains a future direction. The frozen language model, while stable, is not specialized for clinical language, and future work could explore adapting it on labeled dementia corpora. The benchmark itself consists of a relatively small test set (48 participants), and the reported results should be validated on larger, multi-site collections. Finally, the source records depend on the correctness of the initial alignment; work on robust word-to-element linking — perhaps through consensus across multiple description attempts — could improve reliability.
What this means for clinical speech analysis
For researchers and clinicians interested in using spontaneous speech as a dementia screening tool, NarraSpace demonstrates that scene-grounded evidence can be extracted and leveraged without retraining the entire language pipeline. The system's modular design — alignment, descriptor computation, frozen-LM feature extraction, source-record retention — means it can be integrated with existing description protocols as an add-on module. The reported accuracy and AUC indicate that event-level features add discriminative information beyond the transcript alone, and the inspectable evidence trail addresses a common barrier to clinical adoption: the need to understand why a classifier made its decision. By presenting quantitative summaries alongside the original speech evidence, NarraSpace enables a more transparent and evidence-based review process, which may facilitate broader acceptance of AI-assisted speech analysis in clinical practice.