Model Interpretation

Model interpretation covers the set of approaches that seek to let a human understand why a machine-learning model produces a decision. Under that umbrella live three ideas worth keeping apart: interpretability, explainability and causality.

Three distinct concepts

Interpretability is a property of the model itself: the degree to which its logic is directly inspectable because it was designed to be transparent, like a linear regression, a decision tree or a set of rules. Explainability (XAI) is different: post hoc techniques that produce an approximate explanation of an already-trained “black box” model, without opening it. And causality is something else again: it refers to cause-and-effect relationships in the world. Neither interpretability nor explainability establishes causality on its own: both describe the model's behavior, not the process that generates the data.

What Shapley values do (and do not) say

A widely used method is Shapley values, popularized by the SHAP tool (Lundberg and Lee, 2017), which distribute a given prediction among the input features according to their contribution relative to a baseline. An important limit is worth stressing: they are importance attributions, not proofs of cause. A high value means that feature pushed the model's prediction a lot, not that it causes the outcome in reality; if the model relies on a spurious correlation, SHAP will attribute importance to it all the same. Drawing causal conclusions requires an explicit causal framework, not an importance attribution.

Methods and their limits

Methods sort along two axes. First, intrinsic—the model is interpretable by design—versus post hoc—LIME, SHAP or saliency maps, applied after training. Second, local—explaining a single prediction—versus global—describing the whole model. Their limits are real: a post hoc explanation may not be faithful to the model it claims to explain, may be unstable—LIME varies with the random sampling—and may create a false sense of confidence. That is why Cynthia Rudin argues that, in high-stakes decisions such as healthcare or justice, it is preferable to use intrinsically interpretable models rather than explain black boxes after the fact.