SyncAI.news, a Varaisys broadcasting
Keeping Parts Honest: Isolated Part Graphs for Faithful Vision Transformer Explanations
BT

Bernard Tomczyk, Gail Collyer-Hoar, Bryan Williams

· 1 min read

ResearcharXiv cs.CV

Keeping Parts Honest: Isolated Part Graphs for Faithful Vision Transformer Explanations

arXiv:2611.22853v1 Announce Type: new Abstract: Part and prototype-based explanations for Vision Transformers (ViTs) present a decision as evidence from individual image regions. However, self-attention mixes information across patches, so part-level features absorb context from the whole image and consequently fail to faithfully reflect the decision. We show that existing interpretable ViT classifiers depend heavily on this leakage: when each explained region is encoded in isolation, their accuracy collapses, and retraining them under isolation recovers only part of the lost accuracy. We propose a classifier that represents each image as a graph of isolated parts. Each part is encoded only from its own patches by a ViT whose last blocks are fine-tuned under isolation, preventing evidence leakage from elsewhere in the image. Parts are discovered without annotation and refined hierarchically through their sub-parts, making them more consistent and stable across images than those of competing methods. Each edge is computed from part pairs, and recovers evidence that attention previously shared across the object without affecting the parts. Matching the image graph to a learned prototype graph per class makes the prediction a direct combination of node and edge similarities, each computed only from the patches it is credited to, so every explanation is faithful by construction. Across 11 fine-grained and general-object benchmarks, our method exceeds the accuracy of existing explainable methods that rely on leakage, while providing explanations that reflect what the model actually uses.

Original source

This story was published by arXiv cs.CV and written by Bernard Tomczyk, Gail Collyer-Hoar, Bryan Williams. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News