Best way of using DINOv2 image similarity in Weavite

Hello Weaviate gurus. Long time no see :frowning: How boring when I-m not using Weaviate :wink:
I started investigating the wonderful world of multimodal similarity hoping to get some work from a nice historical archive.
I played with CLIP and it is nice but in my experience it works okaish for text to image recognition but somehow fails miserably when asking for an image to image similarity (perceptual+semantic).

After some investigation I think I will want to test the CLIPv2 model.

Not sure as to the best way to use that aside from a ‘bring your own vectors’ approach.

Is there a smarter way?

Thanks a lot

Ciao amico mio! È passato così tanto tempo dall’ultima volta che ci siamo sentiti!

I am pretty sure you will find this interesting, then:

and, of course, source code here: GitHub - antas-marcin/cliparena-page: Website comparing find similar, image and text search using AI CLIP models

Enjoy!

Ciao amico mio. Yes I misssed you too :slight_smile: Boring work for monhts :slight_smile:

As always you are super helpful (and friendly) :slight_smile:

Will investigate.

Hope all good with you and wow the ‘ask AI - internal RAG we now have on the Weaviate forum is soo damn useful. GREAT job.

Take care.

Bob

1 Like

Thanks for sharing this. I was also comparing different approaches for image similarity and hadn’t come across CLIP Arena before. It looks like a useful way to evaluate different CLIP models before deciding what to use in a production pipeline. I’m curious whether anyone here has compared CLIP2 or DINOv2 on real-world datasets where image-to-image similarity is more important than text-to-image retrieval. I’d love to hear about practical experiences.

DINOv2 handles fine-grained visual details much better than CLIP, making it ideal for tasks like historical archives or duplicate image detection.

Bring-your-own-vectors is a perfectly good route here. Weaviate has no built-in DINOv2 module, so you’d define the collection with Configure.Vectorizer.none(), compute embeddings yourself and insert them with the vector parameter. For image-to-image search you can then run near_vector with an embedding of the query image. DINOv2 tends to do well on purely visual similarity, while CLIP is stronger when you also need text queries, so it can be worth keeping both as named vectors.

For the DINOv2 side, the main practical gotcha is consistency: index and query images should go through the same model variant, input resolution and normalization, otherwise cosine scores drift. I’d start with the CLS token from one DINOv2 size and store it as a named vector; if fine-grained duplicate detection still misses, experiment with patch-token pooling separately. Keeping CLIP and DINOv2 as separate named vectors is sensible.