Explainability and Transparency in AI

Explainability in artificial intelligence is the ability to justify a system's individual decisions in understandable terms, especially when its inner workings are opaque —the so-called «black box» of deep neural networks. It is one of three demands that hurried popular writing tends to blur together, alongside interpretability and transparency.

They are not synonyms. Interpretability is a property of the model itself: that its structure is simple enough to follow its reasoning without added tools. Transparency works at the level of governance: documenting the data, the design, the limits and the accountability of a system so that outsiders can audit it.

Explainability, interpretability and transparency: three different things

The distinction matters because each is achieved differently and none implies the others. A model can be explainable without being interpretable: explanations are bolted on after the fact while its internal logic stays put. It can be transparent without being interpretable: its documentation is public even though its computations remain inscrutable. And a small decision tree is interpretable from the root, needing no external explanation at all. Treating the three as one leads to the belief that publishing an explanation amounts to understanding the model —precisely the error the technical literature warns against.

Post-hoc methods versus interpretable models

When the model is already a black box, practitioners turn to post-hoc methods, which explain decisions after training. LIME, proposed by Ribeiro, Singh and Guestrin in 2016, approximates the model's behaviour around a specific prediction with a local linear model. SHAP, from Lundberg and Lee (2017), distributes each feature's contribution to a prediction using Shapley values from game theory. In vision, saliency maps highlight the most influential pixels.

The alternative is not to use a black box at all. Intrinsically interpretable models —decision trees, rule lists, linear models— are readable by design. Cynthia Rudin argued in Nature Machine Intelligence (2019) that for high-stakes decisions these should be preferred over explaining black boxes, because an approximate explanation is not the same as a model one actually understands, and the interpretable option often loses no accuracy.

Transparency, governance and regulation

Transparency has become a documentation practice. Model cards, proposed by Mitchell and colleagues in 2019, summarise a model's intended use, its evaluation conditions and its performance broken down by group. Datasheets, from Gebru and colleagues, do the same for datasets: origin, composition and recommended uses. On the regulatory side, the European Union's AI Act imposes transparency duties —such as disclosing that a user is interacting with an AI, or marking synthetic content— that apply from August 2026. The GDPR, for its part, requires «meaningful information about the logic» of automated decisions in Articles 13(2)(f), 14(2)(g) and 15(1)(h) —Article 22 covers the right not to be subject to a fully automated decision, not that disclosure duty—. On whether it grants a genuine «right to explanation» of individual decisions, the Court of Justice of the EU settled the question in its judgment C-203/22 of 27 February 2025: data subjects are entitled to an intelligible explanation of the procedure and principles applied to their case —handing over the raw formula or algorithm is not enough, and trade secrecy cannot be used to withhold it.

Limits: explanations that are not faithful

The most uncomfortable point: a post-hoc explanation can be unfaithful —it may not reflect the model's actual reasoning. Slack and colleagues showed in 2020 that LIME and SHAP can be fooled into hiding a classifier's bias. Adebayo and colleagues (2018) found that many saliency maps barely change even when a network's weights are randomised, a sign that they do not capture its true decision cues. The danger is false confidence: a plausible but misleading explanation persuades more than no explanation at all. None of this is settled; measuring when an explanation is faithful remains an open problem.