Interpretability in Machine Learning: How to Understand and Explain Models
Machine learning (ML) has advanced by leaps and bounds over the last decade, marking a turning point in how machines learn and make decisions. A critical are...
When a model decides whether you get a loan, whether your CV passes the first filter, or which treatment you are offered, the question that matters stops being «is it accurate?» and becomes «why did it decide that?». That is the terrain of interpretability, and it is worth understanding because concrete rights of yours come out of it.
Two ways to understand a model
At one end sit intrinsically interpretable models: a decision tree or a linear regression can be read directly, because their structure is the explanation. At the other, black boxes — deep neural networks, ensembles of thousands of trees — whose inner workings cannot be followed by eye.
For the latter, post hoc techniques are used: methods applied after training that try to reconstruct why the model answered as it did. And there lies the trade-off that organizes the whole field: the most powerful models tend to be the least readable.
The two techniques worth knowing
LIME, introduced by Marco Ribeiro, Sameer Singh and Carlos Guestrin in 2016, works by local approximation: rather than explaining the whole model, it builds a simple model that resembles it only in the neighbourhood of the specific prediction you want to understand. Hence the name — local, interpretable, model-agnostic explanations.
SHAP, by Scott Lundberg and Su-In Lee (2017), tackles the same problem with a different tool: it distributes responsibility for a prediction among the input variables using Shapley values, a concept from cooperative game theory that answers how to fairly divide the result of a collective effort. Here the «collective effort» is the prediction and the «players» are the variables. The paper unifies six previously separate methods under that framework.
Alongside them sit models designed to be readable from the start, such as generalized additive models and their variants with interactions, and visualization techniques like t-SNE or UMAP, which do not explain the decision but help show how the data are organized.
The objection worth taking seriously
All of the above assumes the way forward is explaining black boxes. One important voice argues the opposite, and a piece on interpretability that omits it is incomplete.
Cynthia Rudin published in 2019, in Nature Machine Intelligence, a paper whose title is already the thesis: «Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead». Her argument: «trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practices». And her conclusion: «There is a way forward — it is to design models that are inherently interpretable».
The nuance matters. A post hoc explanation is an approximation of the model's behaviour, not the model. It can be good and still fail to capture what actually happened in a particular decision. When what is at stake is a sentence, a loan or a diagnosis, Rudin argues that margin is not acceptable.
What the law actually says
There is a widespread error here worth undoing, because almost all coverage repeats it: the General Data Protection Regulation does not grant a «right to explanation» in its articles. That phrase comes from the recitals and from scholarly debate, not from the binding provisions.
What Article 22(1) says is something else: every data subject has the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning them or similarly significantly affects them.
That is: the right is not to be decided by a machine alone. And when two of the foreseen exceptions apply — the decision being necessary for a contract, or explicit consent — paragraph 3 obliges the controller to guarantee at least three concrete things: human intervention, the ability to express one's own point of view, and the ability to contest the decision.
A practical note for anyone wanting to check: the official European portal EUR-Lex serves empty pages to automated readers, so the route that does work is the Spanish official gazette's PDF, which reproduces the full Official Journal text. Article 22 takes two minutes to read.
Explaining is not auditing
There is a confusion worth undoing, because it decides what can be demanded of a system. Explaining means answering for one specific decision; auditing means checking how the system behaves overall. They are not the same work and do not answer the same question.
A local explanation — what LIME or SHAP provide — says which variables weighed in your case. That is what someone contesting a decision needs. But it says nothing about whether the model treats one group of people worse than another: that only shows up by looking at many decisions at once, comparing outcomes across populations. A system can give impeccable explanations one by one and be systematically unfair.
The three rights, in practice
Back to Article 22: when the decision is solely automated and falls under two of the exceptions, paragraph 3 requires guaranteeing human intervention, the ability to express one's point of view, and to contest. It is worth seeing what those are in a real complaint.
Human intervention means a person with authority to change the outcome reviews the case — not that someone signs off what the system already decided. Expressing your point of view means being able to supply information the model did not have: undeclared income, a circumstance the form never contemplated. And contesting means a procedure exists that produces an answer, not a suggestion box.
All three are enforceable and none requires understanding how the model works. That is their virtue: they function even when the system is a black box, which is precisely the situation in which you need them.
Where this applies
In credit risk, interpretability lets whoever assesses an application explain why it is approved or refused — and, on the other side, lets whoever receives it contest with arguments. In medical imaging, visualizations of the regions that weighed most in a classification help the clinician judge whether the model's suggestion makes clinical sense or whether it fixed on an artefact.
In both cases, what interpretability provides is not automatic trust: it is the possibility of disagreeing on solid ground.
The capability: what to ask when a machine decides about you
1. Ask whether the decision was solely automated. That is the condition that triggers Article 22. If there was real human involvement — not a rubber stamp — the framework differs.
2. Ask for the three concrete rights, not «an explanation». Human intervention, expressing your point of view, contesting the decision. They are in the binding text and are enforceable; «explain it to me» is vaguer and easier to brush off.
3. Distinguish explanation from workings. If you are given a post hoc explanation, you are being given an approximation of why the model might have decided that, not a record of what it did. Asking which of the two it is puts you ahead of most people using these systems.
The deep end
- Ribeiro, Singh and Guestrin (2016), LIME.
- Lundberg and Lee (2017), SHAP. The connection to Shapley values is developed in the body of the paper, not in its abstract.
- Rudin (2019). The full title is part of the argument: truncating it loses the thesis.
- Regulation (EU) 2016/679 in the Spanish gazette's PDF. Article 22 sits in the data subject rights section.
This article was produced with artificial intelligence under human editorial oversight.
