Hello Weaviate gurus. Long time no see How boring when I-m not using Weaviate
I started investigating the wonderful world of multimodal similarity hoping to get some work from a nice historical archive.
I played with CLIP and it is nice but in my experience it works okaish for text to image recognition but somehow fails miserably when asking for an image to image similarity (perceptual+semantic).
After some investigation I think I will want to test the CLIPv2 model.
Not sure as to the best way to use that aside from a ‘bring your own vectors’ approach.
Thanks for sharing this. I was also comparing different approaches for image similarity and hadn’t come across CLIP Arena before. It looks like a useful way to evaluate different CLIP models before deciding what to use in a production pipeline. I’m curious whether anyone here has compared CLIP2 or DINOv2 on real-world datasets where image-to-image similarity is more important than text-to-image retrieval. I’d love to hear about practical experiences.
Bring-your-own-vectors is a perfectly good route here. Weaviate has no built-in DINOv2 module, so you’d define the collection with Configure.Vectorizer.none(), compute embeddings yourself and insert them with the vector parameter. For image-to-image search you can then run near_vector with an embedding of the query image. DINOv2 tends to do well on purely visual similarity, while CLIP is stronger when you also need text queries, so it can be worth keeping both as named vectors.
For the DINOv2 side, the main practical gotcha is consistency: index and query images should go through the same model variant, input resolution and normalization, otherwise cosine scores drift. I’d start with the CLS token from one DINOv2 size and store it as a named vector; if fine-grained duplicate detection still misses, experiment with patch-token pooling separately. Keeping CLIP and DINOv2 as separate named vectors is sensible.