
FS
FNU Shivam, Layth Sliman, Ramana Reddy, Donald Adjeroh, Binod Bhattarai, Prashnna Gyawali
· 1 min read
ResearcharXiv cs.CV
Counterfactual Feature Steering for Adjacency-Preserving Identity Unlearning
arXiv:2611.20637v1 Announce Type: new
Abstract: Identity unlearning in text-to-image diffusion models is increasingly important for privacy, safety, and ethical deployment, as it aims to remove the model's ability to generate specific individuals. In this work, we study identity unlearning with \emph{adjacency preservation}, where the goal is to remove a target individual's identity while preserving related non-target identities. We propose Counterfactual Feature Steering (CFS), a method requiring one-time SAE training that performs bounded inference-time concept editing in the latent space of a sparse autoencoder attached to the text encoder. CFS derives a target direction by contrasting forget and retain distributions, then applies a per-token projection proportional to each token's current alignment with this direction, yielding counterfactual representations that suppress the target while limiting collateral damage. To evaluate adjacency preservation, we introduce the Adjacency-Weighted Unlearning Score (AWUS), which measures the tradeoff between target erasure and collateral damage, and extend it to face-specific identity similarity using ArcFace. Across 4,000 prompts derived from Celeb-100, spanning 100 celebrity identities, CFS achieves the strongest evaluated tradeoff between identity erasure of target identities and preservation of adjacent and generic concepts, while remaining stable across intervention strengths where other steering-based methods degrade. Independent ArcFace evaluation and a 10,000-caption COCO study further examine identity and general preservation. Quantitative nudity-suppression benchmarks and qualitative artistic-style examples provide evidence of transfer beyond identities on SD~1.4.
Original source
This story was published by arXiv cs.CV and written by FNU Shivam, Layth Sliman, Ramana Reddy, Donald Adjeroh, Binod Bhattarai, Prashnna Gyawali. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


