Marginaliadaily

A heatmap that shows where a model is actually looking

orig. “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization” · Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra

Interpretability Intermediate 3 min read Written, reviewed by Marginalia Editorial
In the margin
Area
Interpretability, Opening the black box — understanding why a model made a particular decision.

When an image model says "dog", this method draws a heatmap over the photo showing which pixels made it decide that.

Deep models are often a black box: they give an answer with no reason. Grad-CAM highlights the parts of an image that pushed the model toward its prediction, producing a rough heatmap over the picture. If the model says "train" but the heatmap lights up the rails instead of the train, you have learned something about how it really works, and where it might be fooled.

Being able to see why a model decided something matters a lot in areas like medicine, where a wrong reason is dangerous even when the answer happens to be right. Grad-CAM made this kind of check simple and popular. It is a practical entry point into interpretability, the study of opening the black box.

Source

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra, Georgia Institute of Technology

We write original plain-language summaries and link to the source. We never republish the paper.

Paste it and we'll explain it even more simply.

Pass all three to earn the “read & understood” stamp (+10 pts).

Member notes Sign in ↗
ME Marginalia Editorial TEAM
We read the full paper and rewrote it in plain language. Leave your own note below.
Leaderboard · this week

Pass quizzes and leave notes to climb your chapter's board. No chapters are running yet, so this one is wide open.

Start a chapter to compete →
Can We Trust AI's Decision-Making Process?
Intermediate