# Multi2vec-clip without storing image

**URL:** <https://forum.weaviate.io/t/multi2vec-clip-without-storing-image/2243>\
**Category:** Support\
**Created:** [May 8, 2024, 2:33pm UTC](https://forum.weaviate.io/t/multi2vec-clip-without-storing-image/2243 "2024-05-08T14:33:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ffleandro](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/ffleandro/32/1463_2.png) [@ffleandro](https://forum.weaviate.io/u/ffleandro)\
**Post date:** [May 8, 2024, 2:33pm UTC](https://forum.weaviate.io/t/multi2vec-clip-without-storing-image/2243/1 "2024-05-08T14:33:33Z")

</div>

### Description

I’m currently trying to store images in weaviate using the `multi2vec-clip` module without actually storing the blob images, I want to keep them in S3 and just use weaviate for indexing and searching.

I’ve successfully achieved this by calling the `multi2vec-clip` container to manually vectorize each image and store it in weaviate using the `withProperties` + `withVector` operators.

However, I had to define at least one `textField` in my schema, otherwise I wouldn’t be able to use this module. I still want to use the module because I don’t want to manually vectorize my prompts and without this module I can’t use the `nearText` or `nearImage` query operators.

I configured the `description` as a `textField` which contains a detailed generated description of each image.

My question is, how is this field being used, if it is at all?  
When I do a query using `nearText`, does it vectorize the prompt and compare against the vector or is it using somehow the `description` field, like a combination of both `vector` + `description`?

Is there a better way to achieve this: manually generating the image vector at import time, but use the module vectorizer for the query prompt?

Should I be using a different module instead?

### Server Setup Information

- Weaviate Server Version: 1.14.1
- Deployment Method: local docker
- Multi Node? Number of Running Nodes: 1
- Client Language and Version: Javascript/Typescript

### Any additional Information

**docker-compose.yml**

```auto
version: '3.4'
services:
  weaviate:
    image: docker.io/semitechnologies/weaviate:1.14.1
    restart: on-failure:0
    ports:
      - "8080:8080"
    environment:
      LOG_LEVEL: "debug"
      QUERY_DEFAULTS_LIMIT: 20
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: "./data"
      DEFAULT_VECTORIZER_MODULE: multi2vec-clip
      CLIP_INFERENCE_API: "http://multi2vec-clip:8080"
      ENABLE_MODULES: "multi2vec-clip"

  multi2vec-clip:
    image: semitechnologies/multi2vec-clip:sentence-transformers-clip-ViT-B-32-multilingual-v1-1.2.7
    ports:
      - 8081:8080

```

**collection schema**

```auto
{
      "class": "StockImage",
      "moduleConfig": {
          "multi2vec-clip": {
              "textFields": [
                  "description"
              ]
          }
      },
      "vectorIndexType": "hnsw",
      "properties": [
        {
          "dataType": [
            "string"
          ],
          "name": "filename"
        },
        {
          "dataType": [
            "string"
          ],
          "name": "url"
        },
        {
          "dataType": [
            "string"
          ],
          "name": "description"
        }
      ]
    }

```

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [May 8, 2024, 6:22pm UTC](https://forum.weaviate.io/t/multi2vec-clip-without-storing-image/2243/2 "2024-05-08T18:22:36Z")

</div>

hi @ffleandro ! Welcome to our community 🤗

There is a feature request that I believe it suits this use case:

> <https://github.com/weaviate/weaviate/issues/2779>
>
> Hi team,
> I would like to add a new type to Weaviate: \`ImageRef\`.
> 
> \## The prob…lem
> Currently, when using image vector search, we need to store a base64 representation of each image in a \`blob\` property.
> However, in most databases, you rarely would want to query your data and want the database to return the image object (which also makes the returned object quite large).
> 
> Also, implementing an image-centric search is not super straightforward, and we could make it easier to achieve.
> 
> \## The solution
> Could we add a datatype \`ImageRef\` that would be used as a way to find and vectorize an image, without having to store the base64 value?
> 
> 1. ImageRef would store a path or URL to an image resource.
> 2. but also, when used with an image vectorizer (like \`img2vec\`), Weaviate would grab the image, convert it to base64 and send it to calculate a vector – at the end, the base64 value would be discarded. 
> 
> \## Public vs Internal images
> 
> There are two scenarios that we would need to handle.
> 1. Public Images – where the path to the image is accessible from the Weaviate
> 2. Private Images – where the path to the image is \*\*not\*\* accessible from Weaviate
> 
> \### Public
> 
> For images that are accessible with a public URL (for example \[https://weaviate.io/assets/images/hero-6634501914b01d7d336e547ec7ce482c.png\](https://weaviate.io/assets/images/hero-6634501914b01d7d336e547ec7ce482c.png)) – we would only need to pass the image \`path\`, and Weaviate would take care of base64 conversion and vectorization.
> 
> \`\`\`
> data\_to\_insert = {   
> 'title': 'Harry Potter',
> 'text': 'A book about kids trying to do magic',
> 'cover': {
> 'path': 'https://my-bookshop/bookcovers/harrypotter1.jpg'
> }
> }
> \`\`\`
> 
> \### Private image
> 
> For images that are \*\*not\*\* accessible with a path (for example, \`/folder/my-image.png\`), we could pass in the \`path\` and the \`base64\` value.
> 
> \`\`\`
> img\_base64 = convert\_to\_base64(base + '/bookcovers/harrypotter1.jpg')
> 
> data\_to\_insert = {   
> 'title': 'Harry Potter',
> 'text': 'A book about kids trying to do magic',
> 'cover': {
> 'path': '/bookcovers/harrypotter1.jpg',
> 'base64': img\_base64
> }
> }
> \`\`\`
> 
> Alternatively, we could add the base64 conversion to the client.
> 
> \`\`\`
> img\_base64 = convert\_to\_base64(base + '/bookcovers/harrypotter1.jpg')
> 
> data\_to\_insert = {   
> 'title': 'Harry Potter',
> 'text': 'A book about kids trying to do magic',
> 'cover': {
> 'path': '/bookcovers/harrypotter1.jpg',
> 'base64\_convert': True
> }
> }
> \`\`\`

Please, consider leaving a thumbs up so we can measure it’s popularity and move it to be a planned feature in [our roadmap](https://weaviate.io/developers/weaviate/roadmap)

Also, there are some internal discussion around a configuration/way that could avoid having the blob. It is needed - at least for now - as if you change a vectorizable text, a new vector will be produced…

As you were able to manually vectorize your images/texts and provide the vectors while inserting the object, Weaviate will not vectorize the image for you.

One thing you are probably missing here, while generating your own vectors: Whenever you have texts and images properties, Weaviate will pass all the vectorizable properties to the inference model ([like in here](https://github.com/weaviate/multi2vec-clip-inference/blob/131fcbdc1e839d3d3b6968356ae79fdef7b9fa01/smoke_test.py#L39)) and once it receives those back, it will combine all those vectors, [even taking into account some configurable weights](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/multi2vec-clip#collection-level), as you can see [with this code here](https://github.com/weaviate/weaviate/blob/fecf237adbf5d9ca57b9fe9038e9e1c58818fa99/modules/multi2vec-clip/vectorizer/vectorizer.go#L114)

With that **weighted combined vector** and having the multi2vec-clip vectorizer configured, whenever you do a `nearText` or `nearImage` query, Weaviate will vectorize that prompt and match it against the vectors your have and produce close results to all modalities.

As you probably has only vectorized the image, and not combined those with with your texts vectors, your `nearText` will probably not return close results to your objects 🤔

AFAIK, this is the best approach for this case, where you don’t want to store the blob in Weaviate, while we don’t implement the aforementioned feature request.

However, as mentioned, if you provided only an image vector, you will only be able to perform `nearImage` queries.

Let me know if this helps 🙂

Thanks!

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [May 8, 2024, 6:25pm UTC](https://forum.weaviate.io/t/multi2vec-clip-without-storing-image/2243/3 "2024-05-08T18:25:32Z")

</div>

by the way, 1.14.1 is a **really old version**.

We strongly recommend upgrading to a newer version 🙂