Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP

The Reevaluation

Previous Concern (Fig. 4a-c)

Two image embeddings with same distance $r$ to a text embedding...

(a) ...can be close together

(b) ... or far apart

(c) Previous conclusion: a degree of freedom remains → image-image misalignment possible. Image embeddings can lie on any arbitrary points on the circumference, leaving a degree of freedom for intra-modal miscalibration.

Our Explanation (Fig. 4d-f)

(d,e) The two different configurations in (a) and (b) are not arbitrary, but have a good reason to exist: Images in (a,d) and (b,e) have equal distance \(r\) to the "cat" text, but the two images in (a,d) are much more similar to each other than those in (b,e). Displayed distance values are real measurements.

(f) Intra‑modal similarities are a consequence of inter‑modal similarities – no extra degree of freedom. The previous line of argumentation overlooks that each image embedding is bound to more than one text after training.

Previous Concern (Fig. 2)

Figure 2 (by class). Pairwise cosine similarity distributions. Left: Similarities between same class (blue) and opposite class (orange) image feature pairs. A high overlap ratio between the two colors was previously highlighted as an indicator for an intra-modal misalignment issue in CLIP.

Figure 2 (by modality). Similarity distributions of image-text pairs (purple) versus image-image pairs (green). Because CLIP is only supervised on the former, the divergence has previously prompted concerns about whether the latter reflect true similarities.

Our Finding (Fig. 5)

SigLIP (inter-modal only)

SigLIP2 (with intra-modal objective)

Figure 5 (by class). Cosine similarity histograms by class. The distributions are almost equal for purely text-image trained SigLIP (left) and SigLIP2 (right) which includes an image-image self-supervised objective as in the DINO line of work. This indicates intra-class variation is no sign of misalignment brought by pure text-image training, but rather normal behavior.

SigLIP (inter-modal only)

SigLIP2 (with intra-modal objective)

Figure 5 (by modality). Cosine similarity histograms by modality. The divergence in similarity distributions between inter-modal (image-text) and intra-modal (image-image) pairs is nearly identical for both models. The fact that SigLIP2's additional image-image training does not close this gap demonstrates this is not a misalignment introduced by a missing image-image objective.

Previous "Workaround" (Fig. 3r)

Figure 3 (r). Motivated by the intra-modal misalignment hypothesis, a previous work posited it is necessary to convert image-image comparison into image-text comparison.

Our "Back to the Basics" (Fig. 3l)

Figure 3 (l). We go back to the basics.

Questioning Few-Shot Metrics on Toy Dataset

Table 1. Repeating the demonstrative experiment in [1] on the simplistic Dogs vs Cats legacy dataset, where near-perfect results are expected. It was suggested in [1] that poor results with CLIP image-image similarities evidences a misalignment in the image-image space. This hypothesis gets no evidence when swapping model for the uni-modal DINO, a widely acknowledged state-of-the-art image embedder. CLIP scores highest, suggesting that the observed low metrics are not caused by a model weakness and hence neither by a misalignment. Instead, the performance gap between text-image (T-I) and image-image (I-I) can be attributed to the ambiguity in the way the task is conveyed to the model: Two images with opposite labels might still share enough other concepts to be justifiably similar.

Reevaluating Image-to-Image Few‑shot Classification

Table 2. SigLIP (inter‑modal only) outperforms DINOv2, showing no disadvantage from missing intra‑modal loss.

Reevaluating Image‑to‑Image Retrieval

Table 3. Simple $PCA^\leftarrow$ consistently outperforms Optimization-based Textual Inversion (OTI) – no need to convert images to pseudo-text tokens to “fix” a misalignment.

Take Home Messages

Cite this work

@article{herzog2026reevaluating,
      title   = {Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP},
      author  = {Jonas Herzog and Yue Wang},
      journal = {arXiv preprint arXiv:2603.16100},
      year    = {2026}
    }