A heatmap that shows where a model is actually looking

orig. “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” · Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra

Interpretability Intermediate 3 min read Written, reviewed by Marginalia Editorial

In the margin

Heads up

When an image model says "dog", this method draws a heatmap over the photo showing which pixels made it decide that.

What's going on

Deep models are often a black box: they give an answer with no reason. Grad-CAM highlights the parts of an image that pushed the model toward its prediction, producing a rough heatmap over the picture. If the model says "train" but the heatmap lights up the rails instead of the train, you have learned something about how it really works, and where it might be fooled.

Why it matters

Being able to see why a model decided something matters a lot in areas like medicine, where a wrong reason is dangerous even when the answer happens to be right. Grad-CAM made this kind of check simple and popular. It is a practical entry point into interpretability, the study of opening the black box.

Source

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra, Georgia Institute of Technology

View on arXiv ↗ Read the PDF ↗

Still fuzzy on a sentence?

Paste it and we'll explain it even more simply.

Test your understanding

Member notes Sign in ↗

ME Marginalia Editorial TEAM

The editorial team picked this paper because it is worth your time, and an AI drafted this note. We check each one and rewrite it when the draft misses the mark.