Intro to eXplainable AI
ML models drive crucial decisions in AmI (healthcare diagnosis, smart-city management, energy choices, helping data scientists improve a model). In some domains it is not only important to know what is predicted, but also why.
- Knowing the "why" helps in learning more about the problem, the data, and the reason a model might fail.
- XAI techniques unveil the reasoning behind the predictions and decisions of ML models — big in Computer Vision and NLP, but also crucial for time-series sensor data in AmI.
A typical explanation framework
An XAI system takes the current task and makes a recommendation, decision or action; an explanation interface provides an explanation that justifies it; the user then makes a decision based on the explanation. The shift "today → tomorrow": from "Why did you do that? When do you fail? Can I trust you?" to "I understand why; I know when you'll fail; I know when to trust you."
The target of explanations
| Target | Slide definition |
|---|---|
| Non-expert end-users | Take advantage of AI models in daily life without knowledge about ML (e.g., smart-city operators, clinicians, monitored subjects) |
| Technicians | Data scientists / ML experts: monitor the model to understand if the sensing setup needs refinement, if more labeled data are needed, if the model must be improved |
Global vs Local · Accuracy vs Interpretability
Global vs Local explanations
Global explainability
- How does the trained model make predictions? Which features are important in general and how do they interact?
- Usually prohibitive to understand the whole model, but sometimes a portion can be explained (e.g., specific weights)
Local explainability
- Why did the model make a certain prediction for an instance? Locally, the prediction might depend only on some features
- Most widely adopted since it is more accurate than making global explanations
Accuracy vs Interpretability
There is a well-known trade-off: the most accurate models tend to be the least interpretable. From low to high interpretability (and high to low accuracy): Deep Learning → Ensembles (RF, BRT) → SVM → k-NN → Decision/Regression Trees → Lasso/Ridge → Linear Regression.
The three categories of XAI approaches
Interpretable Models
Classic ML algorithms that are inherently explainable (§3).
Model Agnostic
Decouple the explanation from the model, treating it as a black box (§4).
Deep Explanation
Neural networks designed with interpretability in mind (§5).
Interpretable Models
Some classic ML algorithms are inherently explainable: once trained, their parameters can deliver explanations (global or local, depending on the model). Pro: very simple approach. Con: for certain tasks classic ML models are too simple compared to deep learning ones.
Linear Regression
- Prediction = a weighted sum of the feature inputs (y = β₀ + β₁x₁ + … + βₚxₚ + ε).
- The weights β₁…βₚ indicate feature importance → used as a global explanation (high weight = most important feature).
- Pros: simple weight estimation, easy to interpret. Cons: linearity can't capture feature interactions; low predictive performance.
Decision Trees
- Capture complex relationships among features; give both global and local explanations:
- Global: feature importance, by measuring at each node how much it reduced the variance compared to the parent node.
- Local: tracking the path of the data point from root to leaf — the combination of traversed conditions is the local explanation.
Rules Induction
- Learning IF-THEN rules from data; each rule covers a portion of the dataset.
- The set of rules used for a prediction is the local explanation; all the rules together are the global explanation.
- Approaches: Sequential Covering, Bayesian Rules Lists (also iterative approaches, pattern mining).
Model-Agnostic Approaches
Interpretable models are sometimes too simple for complex domains (CV, NLP, time series). Model-agnostic XAI decouples the explanation from the ML model, so it can be applied to any model treated as a black box.
- Model flexibility: the interpretation method works with any model (random forests, deep nets).
- Explanation flexibility: you can choose the best interpretation model based on the goal.
LIME — Local Interpretable Model-agnostic Explanations
LIME analyzes how the model's output changes by varying the input (goal: which parts of the input are important for classification). Given a sample to explain:
The resulting explanation should accurately approximate the local prediction (local fidelity).
Counterfactual Explanations
- Idea: change feature values (realistically), analyze how the prediction changes, observe when it changes relevantly (e.g., the predicted class flips).
- The counterfactual explanation is the feature vector with the smallest change that modifies the prediction (e.g., loan: "if he had been 2.5 years older and requested $210 less for two months shorter, he would have been eligible").
- Pros: clear, easy to implement. Cons: for each instance there may be multiple, possibly contradictory counterfactuals — choosing the best is challenging.
Neural Network Interpretation & Deep Explanation
Interpreting Neural Networks
Methods to visualize features/concepts learned by a network, to explain individual predictions (local). A single deep prediction can involve millions of operations — impossible for humans to follow the exact mapping. Idea: networks learn features in their hidden layers, and the gradient can implement interpretation methods — more computationally efficient than model-agnostic methods looking at the model "from the outside".
GradCAM — Gradient-weighted Class Activation Map
- Algorithm: forward-propagate the input; get the raw score for the class of interest (last neuron before softmax); set other class activations to zero; back-propagate the gradient to the last conv layer; weight each feature-map "pixel" by the gradient for the class.
- Heatmap: ReLU(Σₖ αₖᶜ Aᵏ) where the weights αₖᶜ are a global average pooling of the gradients δyᶜ/δAᵏ; ReLU keeps only positive values.
Deep Explanation: prototype-based networks
Posthoc methods (chosen architecture first, then interpret it) have problems: explanations change a lot based on the model used to generate them and may not make sense to humans. Deep explanation instead designs networks interpretable by design.
- Prototype-based = case-based reasoning: explain predictions by similarity to prototypical cases. A prototype is a data point close/identical to a training observation; a limited set should represent the whole dataset.
- A self-explaining model that learns prototypes has three components: an autoencoder (latent space), a prototype layer (learns m prototypes), and fully-connected + softmax layers for classification. It learns in parallel the latent space, the classification task, and the prototypes.
- Distance to each prototype = squared L² distance in latent space; explanation = the k closest prototypes. Two regularization terms: each prototype close to ≥1 training example, and every training example close to ≥1 prototype. To visualize latent prototypes, use the Decoder to reconstruct them.
Evaluating Explanations & XAI for Time Series
How to evaluate explanations
It's tempting to think convincing explanations are also effective — but convincing explanations may be associated with model mistakes. A key challenge is over-reliance: the user trusts the system too much just because it provides explanations.
| Measure | What it assesses |
|---|---|
| User Satisfaction | Clarity / utility of the explanation — usually via questionnaires |
| Mental Model | How well a user understands the underlying model — by asking users to describe the explanation process |
| Trust Assessment | Trust is the cognitive factor influencing perception (can be positive or negative) — questionnaires before/after using the model |
| Correctability | Whether XAI can identify errors (leading to correction and continuous training) — usually assessed with automatic methods |
XAI for time series
| Explanation type | How it explains |
|---|---|
| Time-points based | Assign a relevance score to every time point (how much each sample contributes). Attribution methods (external, e.g., LIME, GradCAM) vs Attention methods (internal focus of the network) |
| Subsequence based | Identify sub-parts of the series responsible for the outcome — by extracting motifs (repeating patterns) and training interpretable models on them |
| Instance based | Use the whole temporal window: feature-based, prototype-based, or counterfactual-based (minimal change flipping the outcome; minimality uses a distance between time series) |
AmI use case 1 — DeXAR (explainable smart-home HAR)
Raw sensor measurements are hard to explain to non-expert users, so DeXAR works on semantic states (high-level info describing what happened in a time interval — e.g., a pressure mat ON→OFF becomes using_kitchen_chair [t1,t2]). Pipeline:
- A semantic image includes the semantic states observed in the segment + the most recent K past activities; any CV XAI method (LIME, GradCAM, prototypes) can be applied.
- The heatmap is useful for data scientists; since each pixel has a semantic, a natural-language explanation is built (threshold the relevant features → sentence via heuristics or an LLM).
Explanation Score
A metric to automatically evaluate the consistency of an explanation with common-sense knowledge about the HAR domain (e.g., washing dishes happens in the kitchen, after eating). Common-sense knowledge is encoded in a semantic model defining which features partially explain each activity; a feature counts positively if it partially explains the activity, negatively otherwise.
AmI use case 2 — X-CHAR (counterfactuals for nurse care)
When nurses only wear an inertial device, complex activities are recognized as sequences of simpler low-level activities ("concepts"). X-CHAR learns to recognize concepts and then complex activities end-to-end, and produces a counterfactual explanation: it stores all concept sequences from training, and for a test sequence c finds the nearest one cex with minimum distance but a different class (e.g., "it would be 'Cleaning the Patient' had the sequence been Measure Blood Glucose → Oral care → Cleaning Genital Area").
Final Quiz — Exam Style
15 MCQs + 2 open-ended. Two are the actual simulation questions (Q3, Q21) reproduced verbatim; the rest follow the professor's recipe — sibling distractors, one-word swaps, and a FALSE question hiding an absolutizer.
Which of the following is a "counterfactual explanation"?
Which information of the neural network does GradCAM use to generate a heatmap highlighting the regions of the input considered important for classification?
a) XAI techniques unveil the reasoning behind ML predictions — important because in many AmI domains (healthcare, smart cities, energy) it matters not only what is predicted but why, helping understand the problem/data and why a model may fail; HCI and psychology matter too, since explanations must be effectively delivered to the target (non-expert end-users vs technicians). Global explainability asks how the trained model predicts in general and which features matter overall (usually prohibitive for the whole model); local explainability asks why a specific instance got its prediction (most adopted, more accurate than global). Accuracy–interpretability trade-off: the most accurate models are the least interpretable — Deep Learning (high accuracy, low interpretability) → Ensembles → SVM → k-NN → Trees → Lasso/Ridge → Linear Regression. Three categories: Interpretable Models, Model-Agnostic, Deep Explanation.
b) Linear regression: prediction = weighted sum of features; the weights = feature importance → a global explanation (simple, but can't capture interactions and has low accuracy). Decision trees: capture complex feature relationships; global = feature importance from how much each node reduces variance vs its parent; local = the conditions on the root-to-leaf path of the data point. Rules induction: learn IF-THEN rules, each covering a portion of the data; the rules firing for a prediction are the local explanation, all rules together the global one (e.g., Sequential Covering, Bayesian Rules Lists). Pro: very simple; con: classic models are sometimes too simple versus deep learning.
a) Model-agnostic XAI decouples the explanation from the model (any black box). LIME: generate perturbed versions of the input, predict them with the black box, and fit an interpretable surrogate (e.g., linear regression) with samples weighted by similarity to the original, achieving local fidelity. Counterfactual: "if X had not occurred, Y would not have occurred" — the smallest realistic change to the input that flips the prediction (pros: clear, easy; cons: multiple, possibly contradictory counterfactuals). Problem of model-agnostic: explanations don't always mimic the model's real calculations and may use different features → motivates explaining deep models directly. GradCAM: for CNNs, back-propagate the class score's gradient to the last convolutional layer, global-average-pool the gradients as weights, ReLU(Σ αₖᶜAᵏ) → a heatmap of relevant input regions. Prototype-based deep explanation: case-based reasoning; a network learns in parallel a latent space (autoencoder), a prototype layer (m prototypes), and the classifier; explanation = the k closest prototypes (squared L² distance), visualized via the decoder; two regularizers tie prototypes and training examples together.
b) Evaluation: convincing ≠ effective — a key risk is over-reliance (trusting the system just because it explains). Measures: User Satisfaction (clarity/utility, questionnaires), Mental Model (does the user understand the model — describe the process), Trust Assessment (cognitive, positive or negative, questionnaires before/after), Correctability (can XAI identify errors → correction & continuous training, automatic methods). Time series: time-points based (relevance per sample — attributions like LIME/GradCAM vs internal attention), subsequence based (motifs), instance based (feature/prototype/counterfactual). DeXAR: extract semantic states → semantic image → explainable DL classifier + CV XAI heatmap → natural-language explanation. Explanation Score: automatically measures consistency with common-sense knowledge (which features partially explain an activity). X-CHAR: recognizes complex nurse activities as sequences of concepts and gives a counterfactual = the nearest training concept-sequence with minimum distance but a different class.