# What is Explainability?

Also called Interpretability, XAI, Explainable AI.

Explainability is the degree to which the reasons for an AI system's output can be presented in terms a person can understand and check. It matters where decisions affect people, where errors must be diagnosed, and where regulation or contract requires a rationale. For large neural networks it remains only partially achievable.

Two related terms are used inconsistently across the field. Interpretability usually refers to a model whose internal workings can be inspected directly, such as a small decision tree or a linear model with few features. Explainability usually refers to producing an after-the-fact account of a decision made by a model too complex to inspect. The boundary is blurry and some authors reverse the usage.

Post-hoc techniques include attributing an output to input features, measuring how the output changes when inputs are perturbed, surfacing which retrieved documents most influenced a response, and presenting counterfactuals showing what would have changed the outcome. A separate line of mechanistic interpretability research attempts to reverse engineer internal circuits and features, which is promising and far from complete for large models.

Language models introduce a specific trap. When a model states its reasoning, that text is generated by the same process as the rest of the output and is not a log of the computation that produced the answer. Research has repeatedly found stated reasoning that does not reflect the factors actually driving a model's behavior, so a plausible explanation should not be taken as a faithful one.

In practice, verifiable grounding often serves better than generated rationale. Citing the specific source passages behind an answer, exposing which tools were called with which arguments, and showing the retrieved evidence lets a person check the basis of a response directly. That is weaker than true mechanistic understanding and considerably more useful than an unverifiable narrative.

## Key points

- Interpretability inspects the model, explainability accounts for a decision
- Techniques include attribution, perturbation, and counterfactuals
- Stated reasoning is generated text, not a log of the computation
- Plausible explanations are not necessarily faithful ones
- Citations and tool traces give checkable grounding instead

## In practice

An assistant answers a policy question and includes a paragraph explaining how it reached the conclusion. The explanation is coherent but cannot be verified. The same system also lists the three policy documents retrieved, with the exact passages quoted and linked. A reviewer can confirm or reject the answer from those passages in seconds, without relying on the narrative at all.

## Related terms

- [Hallucination](/en/glossary/hallucination)
- [AI Governance](/en/glossary/ai-governance)
- [Human in the Loop](/en/glossary/human-in-the-loop)
- [EU AI Act](/en/glossary/eu-ai-act)
- [Model Evaluation](/en/glossary/model-evaluation)

[Back to the AI Glossary](/en/glossary)