SyncAI.news, a Varaisys broadcasting
A Threshold-Free Visual-Semantic Matching Metric for Evaluating Consistency in Subject-Driven Image Generation
BS

Bowei Shan, Heng Yin

· 1 min read

ResearcharXiv cs.CV

A Threshold-Free Visual-Semantic Matching Metric for Evaluating Consistency in Subject-Driven Image Generation

arXiv:2611.19780v1 Announce Type: new Abstract: In subject-driven text-to-image generation, quantitatively evaluating localized visual consistency across varying poses and environments remains a major challenge. Existing metrics either rely on global image embeddings (CLIP, DINO) and Vision-Language Models (VLMs) that pool spatial representations and miss fine-grained textural artifacts, or depend on scalar threshold-gated dense metrics such as Visual-Semantic Matching (VSM) that suffer from point-alignment dropouts, severe threshold sensitivity, and score instability. To overcome these limitations, we propose TF-VSM, a fully continuous, threshold-free visual-consistency metric paired with an extended feature-disentanglement retraining schedule for dual-branch ResNet feature-aggregation networks. Rather than enforcing rigid binary cutoffs $(T_s,T_v)$, TF-VSM weights every object point by a non-linear semantic match confidence $w_j=ReLU(\hat{s}_j)^2$ and continuously integrates visual similarity over all consistency levels $r\in[0,1]$, yielding a closed-form confidence-weighted score without any hand-tuned hyperparameters. On a controlled 100-pair benchmark with ground-truth inpainting oracles, TF-VSM attains Pearson $0.638$ and Spearman $0.674$ oracle correlation, improving over the pretrained VSM baseline ($0.454/0.589$) by $+0.171/+0.116$ after retraining, and drastically outperforming CLIP ($-0.052/-0.004$), DINO ($0.087/0.121$), and VLMs ($0.096/0.003$). Six diagnostic ablations confirm that smooth confidence weighting is the right mechanism, that hard semantic gating is brittle (dropping to $0.533$ Pearson at $T_s{=}0.8$), and that the threshold-free metric is essentially invariant to the training-time loss balance $\alpha$ that the thresholded metric still depends on.

Original source

This story was published by arXiv cs.CV and written by Bowei Shan, Heng Yin. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News