Energy consumption forecasting depends on machine learning models that facility managers and building operators need to understand but often cannot. Post-hoc explanation techniques like SHAP and DiCE exist, but the interfaces that deliver them, traditional point-and-click dashboards, require substantial technical expertise and offer limited flexibility for dynamic inquiry. Rodion Krjutškov, Eduard Barbu, Nikos Sakkas, and Sofia Yfanti from Nupp Software, the University of Tartu, Apintech Ltd, and Hellenic Mediterranean University tackle this problem by building a conversational interface that lets domain experts ask questions in natural language and receive structured explanations from the underlying model.

Why Existing Conversational XAI Systems Fall Short

TalkToModel, the main predecessor to this work, introduced the idea of using natural language dialogue for explainability. But it relied on fine-tuned language models with custom grammars that constrained what users could ask. The grammar-based parsing achieved only 76.8% intent-parsing accuracy, meaning roughly one in four user queries was misunderstood. In practice, this undermines trust and prevents production deployment: a facility manager who asks the wrong question and gets a confident but irrelevant answer will stop using the tool.

The Explainability Assistant takes a fundamentally different approach. Instead of fine-tuning a language model on task-specific grammars, it provides the LLM with JSON specifications of available functions and lets the model's function-calling mechanism determine which functions to invoke and with what arguments. This eliminates the need for task-specific fine-tuning entirely. The system adapts to new datasets and model types through configuration changes alone, without modifying the core application logic.

Architecture: Separating Presentation from Computation

The system uses a modular architecture with a clear separation between front-end and back-end. The front-end is built with Next.js and provides the conversational interface. It manages client-side conversation history, handles user authentication, and lets users select from multiple LLM providers, including Llama-3.3-70B-Instruct, Gemini-2.0-Flash, and Gemini-2.5-Flash. Users can switch between models within a single session.

The back-end is a FastAPI application that orchestrates data operations, runs ML model inference, executes pre-defined explainability functions, and communicates with external LLM providers via API. The function-calling paradigm works as follows: the back-end sends the user query, conversation history, and JSON schemas of available functions to the LLM. The LLM returns structured JSON specifying which function to call and with what arguments. The back-end executes the function locally, retrieves the result, and sends it back to the LLM for natural language synthesis.

This design has four concrete advantages. First, accuracy. Gemini-2.5-Flash achieves 94% exact-match parsing accuracy on gold-standard datasets, compared to 76.8% for TalkToModel's grammar-based approach. Second, transparency. The LLM is constrained to respond with a structured JSON format that includes a mandatory free-form explanation field describing the planned actions, allowing users to verify the system's interpretation before execution. Third, privacy. Function execution happens entirely on the back-end, retrieving results from local data and models. Only the user query and tool schemas are sent to the LLM provider, so proprietary datasets and model internals stay protected. Fourth, deployment flexibility. The system adapts to new datasets and model types through configuration changes alone.

The XAI Toolkit: SHAP, DiCE, and What-If Analysis

The system integrates three established explanation techniques. SHAP provides feature importance analysis grounded in game-theoretic principles, giving consistent and theoretically justified feature attributions. Users can request explanations for individual predictions or global summaries across the dataset. DiCE generates counterfactual explanations, identifying minimal, realistic modifications to input features that would alter the model's prediction. What-if scenario analysis lets users specify precise hypothetical changes and observe their effects on predictions, useful for policy analysis and sensitivity testing.

The function-calling architecture handles these differently from traditional XAI dashboards. Instead of requiring the user to navigate tabs and select explanation methods, the system automatically determines which explanation technique applies to a given natural language query. A question like "Why was sample 42 predicted to have such high consumption?" triggers SHAP. A question like "What would change the prediction to low consumption?" triggers DiCE. A question like "If indoor temperature increases by 5 degrees, how much will consumption change?" triggers what-if analysis. The user never needs to know which technique to select.

Parsing Accuracy: 94% Across Multiple LLMs

The authors constructed three gold-parse datasets for the energy consumption use case. Dataset A, with 20 samples, was created manually to cover core system functionalities across 20 available functions. Datasets B and C, with 80 samples each, were generated using frontier LLMs GPT-5 and Gemini-2.5-Pro with Dataset A as few-shot examples, then manually verified and corrected. Cross-validation on combined sets ensures independence from the generation process.

Four cost-effective LLMs were evaluated: GPT-5-mini, Llama-3.3-70B-Instruct, Gemini-2.0-Flash, and Gemini-2.5-Flash. Gemini-2.5-Flash achieved the highest accuracy at 94% on one combined dataset and 93% on the other, followed by Gemini-2.0-Flash at 88% and 91%. GPT-5-mini and Llama-3.3-70B lagged behind at 74-79%. The gap between the best and worst models shows that function-calling accuracy depends heavily on the underlying LLM's capability, not just the architecture.

The improvement from 76.8% to 94% is not just a marginal gain. In a production setting, a 23-point accuracy improvement means the difference between a system that frustrates users and one that reliably interprets their intent. The authors note that even small parsing errors compound in multi-turn dialogues, where a misunderstood query in turn two can derail the entire conversation.

Expert Evaluation: Unanimous Preference for the Conversational Interface

The validation study involved three energy domain specialists with 10-20 years of experience in building energy management. The study used a within-subjects design, meaning each expert used both interfaces. The two interfaces were the Explainer Dashboard, a traditional point-and-click graphical tool based on the open-source Explainer Dashboard framework, and the Explainability Assistant.

Participants completed ten task questions, five per interface, covering prediction retrieval, feature importance identification, counterfactual reasoning, error analysis, and model accuracy assessment. Task accuracy was 93% for the Explainer Dashboard and 100% for the Explainability Assistant. While the sample size is too small for statistical significance, the consistency of the result across all three experts is notable.

The subjective ratings tell the most compelling story. All three experts rated the Explainability Assistant at or near maximum (5 out of 5) for "Easy to use" and "Would use again." The Explainer Dashboard received variable ratings, with scores ranging from 2 to 5 across experts and dimensions. The most striking contrast appeared in "Would use again," where the Explainability Assistant achieved unanimous 5s while the Explainer Dashboard scored between 2 and 4. The moderate ratings for "Trust outputs" and "Understand system" suggest appropriate epistemic caution rather than uncritical acceptance, which aligns with the design goal of supporting appropriate reliance.

Participants also noted that the Explainer Dashboard's visual overview was occasionally useful for orientation, suggesting that hybrid conversational-visual interfaces may offer complementary benefits. The asynchronous study format, while accommodating participants' schedules, prevented observation of real-time interaction patterns that could reveal deeper insights into query formulation during exploratory analysis.

Adapting to New Domains Without Retraining

To demonstrate architectural modularity, the authors deployed the system on the Heart Disease Cleveland dataset, a classification task with 303 patients and 14 clinical attributes using a Decision Tree classifier. The transition required updating the JSON function specifications only, without modifying the core application logic or fine-tuning the LLMs. The system successfully moved from predicting continuous energy values to explaining binary classification risk factors.

This is the practical significance of the function-calling approach. Traditional conversational XAI systems require task-specific fine-tuning for each new domain, which is expensive and limits deployment breadth. The Explainability Assistant treats explanation methods as a configuration problem, not a training problem. Adding a new explanation technique or adapting to a new model type means writing a new function specification in JSON, not collecting labeled data and retraining a parser.

Limitations and What Comes Next

The expert validation used only three participants, which limits statistical power. The within-subjects design partially compensates by controlling individual differences, but the findings need replication with larger, more diverse user populations. The fixed interface order (dashboard first, then conversational) may have introduced learning effects that inflated ratings for the second interface. Future studies should counterbalance the order.

The evaluation focused on energy forecasting with symbolic regression models. Whether the same usability advantages hold for other domains, especially high-stakes settings like healthcare, requires validation. The authors also acknowledge that larger-scale studies with temporal analyses of extended usage patterns are needed to understand long-term adoption.

The open-source release of the complete system, including front-end, back-end, gold-parse datasets, evaluation scripts, and the full expert validation questionnaire, provides the research community with a concrete starting point for extending conversational XAI to new application domains. The modular architecture means that adding a new explanation technique or adapting to a new model type is primarily a configuration task, not a development task, which lowers the barrier to experimentation.

Read the paper on arXiv