Interpretability

Interpretability is the degree to which you can look inside an AI model and understand what its internal parts represent and how they combine to produce an answer, as opposed to explainability, which constructs a plausible justification after the decision is made.

Safety & governancemechanistic interpretabilitymodel transparency

§ 01DefinitionInterpretability

No jargon, no math

In plain words, what is it?

An AI model is billions of numbers. Interpretability asks: can we say what some of those numbers mean? The research route is mechanistic. You feed the model examples and watch which internal units fire, then attach a meaning to them: a unit that fires on legal Latin, another on incomplete welds, another on a date that is already past. Find enough of these and you can trace a path: this input lit up these units, which pushed the answer towards “reject”. Interpretability opens the box; explainability narrates from outside. An explainer says “the loan was refused because income is low”, which may be true, plausible and wrong. An interpretability result points at the actual units and activations, so it can be checked. Where it breaks: modern models are too large to read end to end, so you get partial views, and a feature you have named can still be doing something else in contexts you did not test. Interpretability raises confidence; it does not issue certificates.

A real example

The mystery. A sheet-metal plant's vision system flags 12% of welds as defective on Tuesdays, triple the weekday average. Retraining on fresh data does not help, so the team stops guessing and opens the model.The look inside. An interpretability tool ranks the model's internal features by how much they drive the Tuesday rejections. One feature dominates: it fires on a yellow-blue colour imbalance in the image.The cause. The Tuesday early shift changed the inspection lamps in January. The new LEDs are 4000K; the old ones were 6500K. The model was never rejecting welds on Tuesdays; it was rejecting the light.The fix. New lamps, plus a lamp-colour check in the capture pipeline. The interpretability report stays in the quality file, next to the calibration logs, as evidence the cause was found in the model, not asserted by it.

§ 02Glossary3

Terms that live nearby.

SAFETY & GOVERNANCE

Explainability

Explainability is the property of an AI system (or of a layer added to it) that lets you see why it gave a specific answer: which inputs mattered, which rules fired, and what would have changed the outcome.

Read

SAFETY & GOVERNANCE

AI Bias

AI bias is a systematic skew in what a model predicts or decides, caused by skewed training data or skewed design, so it treats similar people or cases unfairly and consistently.

Read

SAFETY & GOVERNANCE

Alignment

Alignment is the work of making an AI system do what you actually meant, not merely what you literally typed: its goals, limits and refusals are shaped during training and deployment so its behaviour stays within what operators and society intend.

Read

§ 03

Want this running in your stack?

We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.

Book a scoping call