Essay · Explainable AI · Design
Designing Trust with Explainable AI
On why surfacing the right data isn't enough — and what it actually means to design for trust when an AI makes a call that affects someone's life.
01 / The Moment
I once watched someone refuse to trust an AI that was correct.
She was using Merlin, a bird identification app. The model had correctly identified a warbler. It showed her a heatmap — red blobs highlighting the parts of the image it had used to make its call.
She stared at it for a moment, then shrugged. "I don't really know what I'm looking at." And she moved on, unconvinced.
01 / The Moment
The AI was right.
The explanation existed.
But trust? That didn't happen.
The explanation was technically accurate. The heatmap showed exactly what the model had attended to. Everything worked as designed.
And yet the person standing in front of it left less confident than when she arrived.
01 / The Moment
We treat explainability like a technical problem.
Surface the right data. Show the right chart. Reveal the right confidence score — and people will trust the system.
But trust doesn't work that way. Trust is human. And designing for it means designing for people, not for algorithms.
Merlin Bird ID · Cornell Lab of Ornithology — the heatmap explanation that triggered this essay
The observation
"I don't really know what I'm looking at."
A user, after viewing a correct AI explanation — and still not trusting the result.
02 / The Research
The study that changed how I think about this.
Princeton University researchers observed 20 Merlin Bird ID users — some casual, some with deep AI literacy — and tested four different ways of explaining the model's decisions.
What they found wasn't about which explanation type was best. It was about how differently people needed to be met.
02 / The Research
Four explanations.
Twenty users.
Completely different needs.
The study tested heatmaps, example-based explanations, concept-based explanations, and prototype comparisons — all with the same underlying model.
No single explanation type won. Because the question wasn't about the explanation. It was about the person receiving it.
02 / The Research
Less AI experience: calibrate instinct.
More AI experience: audit the model.
People with less AI experience didn't want to understand the model. They wanted to calibrate their own instincts. Does this feel right? Should I believe it?
People with more AI experience pushed on confidence scores. They wanted to know why the model was uncertain, not just that it was. They weren't looking for reassurance. They were auditing.
02 / The Research
Same app. Same feature.
Completely different human needs.
This is what the research made visible. The surface was identical. The people underneath it were not.
Princeton University research · Merlin Bird ID · 20 participants · 4 explanation types
The insight
Explainability isn't a box you check. It's a relationship you design.
And like any relationship, it requires you to actually know who you're talking to.
03 / The Design Implication
Design for a beginner,
show it to an expert.
You don't build trust — you build suspicion. "What aren't they showing me? Why does this feel simplified?"
The expert needs the seams. They need to see where the model is uncertain, how it was trained, what it optimizes for. Hiding that isn't clarity. It's a red flag.
Design for an expert,
show it to a beginner.
You don't build trust — you build anxiety. "Why is there so much to read? What am I missing?"
The beginner needs orientation, not depth. A highlighted wing, a circled beak, a simple verdict. Enough to feel anchored. Not so much that they feel lost.
04 / Applied to Fintech
Loan approvals. A verdict — or a conversation?
Most people who get rejected don't understand why. They get a score, maybe a vague reason code, and that's it.
What if instead, you got: "Your debt-to-income ratio is 42%. If you bring that below 36%, your approval likelihood increases significantly." Suddenly it's not a verdict. It's a conversation. The AI has handed you something to do.
04 / Applied to Fintech
Investment dashboards. Authority — or partnership?
Instead of the model just telling you what to do, imagine being able to pull levers — change your contribution, adjust your risk level, shift your timeline — and watch how the recommendation responds in real time.
The AI isn't the authority anymore. It's a thinking partner. You bring the context. It brings the model. You figure it out together.
04 / Applied to Fintech
In every case, the explanation was the collaboration.
Expense categorization that explains why it tagged something a certain way — so you can trust it, or correct it. Cash flow forecasts that respond to your inputs. Transfer costs broken down in real time.
The explanation wasn't decoration. It was the thing that made the interaction feel like a partnership rather than a black box handing down a verdict.
Interactive recommendation — contribution lever, risk slider, live AI response
05 / The Part That Stayed With Me
05 / The Reframe
I came into this thinking about transparency. I left thinking about participation.
The users who trusted AI the most weren't the ones who understood it best technically. They were the ones who felt like they were in it — like they had a hand on something, like their input mattered, like the system was responding to them as a person and not just processing their data.
That's what we're really designing for. Not legibility. Not auditability. Partnership.
And it changes the question we should ask ourselves.
Every time we decide how — or whether — to explain what an AI just did, we're making a choice about whose interests we're serving.
Are we designing this explanation to help the person in front of us? Or are we designing it to make the system look trustworthy?
Those are very different things. And the difference shows.
06 / The Question Worth Asking
Are we designing this explanation to help the person in front of us?
Or are we designing it to make the system look trustworthy?
Those are very different things. And I think that's a more honest way to frame the work — every time we decide how, or whether, to explain what an AI just did.
Trust is not a feature. It is not a confidence score. It is not a heatmap. It is a relationship — and relationships require knowing who you are talking to.
← Back to Notes