# I am getting hnsw\_vector\_cache\_prefill frequently

**URL:** <https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501>\
**Category:** Support\
**Created:** [November 8, 2024, 8:43am UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501 "2024-11-08T08:43:47Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\
**Post date:** [November 8, 2024, 8:43am UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/1 "2024-11-08T08:43:47Z")

</div>

### Description

Hello I am getting hnsw\_vector\_cache\_prefill in log very frequently.

 ![image](https://canada1.discourse-cdn.com/flex027/uploads/weaviate/original/2X/8/8a51ffdcd8011d685b1fc2a876ba70134e94c4e5.png)

I don’t information on this log.  
In addition of above, I am also getting Query call with protocol GRPC batch failed with message Deadline Exceeded and also  
 ![image](https://canada1.discourse-cdn.com/flex027/uploads/weaviate/optimized/2X/2/25652e4d7862f7e366ae6630a596669a625fa0ac_2_690x21.png)

Can any one help me ?

### Server Setup Information

- Weaviate Server Version: 1.27.1
- Deployment Method: 
- Multi Node? Number of Running Nodes: 1
- Client Language and Version: Python-latest
- Multitenancy?: Yes

### Any additional Information

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [November 8, 2024, 11:11am UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/2 "2024-11-08T11:11:11Z")

</div>

hi @2020ashish !!

Welcome to our community 🤗

the `hnsw_vector_cache_prefill` is just a `INFO` log, so no worries here.

Can you paste the entire stack trace error for the second issue?

---

<div class="post-metadata">

**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\
**Post date:** [November 12, 2024, 11:26am UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/3 "2024-11-12T11:26:59Z")

</div>

Apologize, for late reply  
It was from OOMKill issue.  
I have few question.  
In our case we have ingest all of data which is semi structured.  
While ingesting there are so many read/ write operation to update previous data.

1. While ingestion, First I check if data exist or not then update certain field. Ingestion time is less compare to updation time. What should be best approach ?

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [November 12, 2024, 12:40pm UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/4 "2024-11-12T12:40:38Z")

</div>

Hi! The best approach is doing a batch with deterministic IDs.

If you want to define a fixed ID for each object, that’s the way to go:

> **[Create objects | Weaviate](https://weaviate.io/developers/weaviate/manage-data/create#generate-deterministic-ids)**
>
> The examples on this page demonstrate how to create individual objects in Weaviate.

Remember to use `batch` instead of `insert` or `insert_many`

Let me know if this helps!

Thanks!

---

<div class="post-metadata">

**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\
**Post date:** [November 12, 2024, 1:13pm UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/5 "2024-11-12T13:13:00Z")

</div>

We are using batch.  
While ingestion we need to check if record exist or not than inserting.  
Sample code provide, how we are ingesting data in some case.  
while updation time is huge for below approach.  
Please suggest any optimized way ?

```auto
class_name = "Book"
if client.collections.exists(class_name):
    client.collections.delete(class_name)
client.collections.create(
    name=class_name,
    vectorizer_config=wcc.Configure.Vectorizer.text2vec_transformers(),
    multi_tenancy_config=wcc.Configure.multi_tenancy(enabled=True),
    inverted_index_config=wcc.Configure.inverted_index(
        index_timestamps = True
    ),
    properties=[
        wcc.Property(
            name="Book_name",
            data_type=wcc.DataType.TEXT,
            tokenization=wcc.Tokenization.FIELD,
        ),
        wcc.Property(
            name="Author",
            data_type=wcc.DataType.TEXT,
            tokenization=wcc.Tokenization.WORD,
            skip_vectorization=True,
        ),
        wcc.Property(
            name="Book_Summary",
            data_type=wcc.DataType.TEXT,
            tokenization=wcc.Tokenization.FIELD,
            
        ),
        wcc.Property(
            name="Update_date",
            data_type=wcc.DataType.TEXT,
            tokenization=wcc.Tokenization.FIELD,
            skip_vectorization=True,
        ),
    ],
)

```

```auto

def check_book_exist(client, class_name, value, tenant):
    filter = wvc.query.Filter.by_property("Book_name").equal(value)
    class_obj = client.collections.get(class_name).with_tenant(tenant)
    response = class_obj.query.fetch_objects(filters=filter, limit=1)
    uuid = None
    for o in response.objects:
        uuid = o.uuid
    return uuid

def update_records(uuid, collection, data, tenant, client):
    class_obj = client.collections.get(collection).with_tenant(tenant)
    class_obj.data.update(
        uuid=uuid,
        properties=data,
    )

def parser(data,tenant):
    with client.batch.fixed_size(batch_size=100) as batch:
        book_uuid = check_book_exist(
            client = client,
        class_name= "Book",
            value = data.get("Book_name"),
            tenant = tenant,
        )
        if book_uuid is None:
            book_data = { "Book_name": data.get("Book_name"),
                        "Author": data.get("Author"),
                        "Book_Summary": data.get("Book_Summary"),
                        "Update_date": data.get("Update_date"),
                }
            batch.add_object(
                properties=book_data,
                collection="Book",
                tenant=tenant,
            )
            print("Added the record for book name", data.get("Book_name"))
        else:
            update_records(
                tenant=tenant,
                client=client,
                uuid=book_uuid,
                collection="Book",
                data={"Update_date": data.get("Update_date")},
            )
            print("Update the record for book name", data.get("Book_name"))

```

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [November 12, 2024, 1:40pm UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/6 "2024-11-12T13:40:02Z")

</div>

Hi!

If you can define an id per book, you can do this:

```python
import numpy as np

from weaviate.util import generate_uuid5
x = 1000
collection_name = "Book"

collection = client.collections.get(collection_name)
with collection.batch.dynamic() as batch:
    for i in range(x):
        batch.add_object(
            {"text": f"{collection_name}, object {i}"},
            vector=np.random.rand(1536),
            uuid=generate_uuid5(f"book-id-{i}")
        )

if collection.batch.failed_objects:
    print("Failed Objects: ", collection.batch.failed_objects)

```

So passing a fixed uuid (generated from a book-id from your system, for example) will make the batch process to insert or update. if there is an object with the same id: it updates or do nothing, and if there isn’t, it creates.

🙂

---

<div class="post-metadata">

**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\
**Post date:** [November 13, 2024, 10:04am UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/7 "2024-11-13T10:04:08Z")

</div>

{“action”:“lsm\_replace\_compacted\_segments\_blocking”,“build\_git\_commit”:“05de0db”,“build\_go\_version”:“go1.22.8”,“build\_image\_tag”:“v1.27.1”,“build\_wv\_version”:“1.27.1”,“class”:“Book”,“index”:“Book”,“level”:“warning”,“msg”:“replacing compacted segments took 343.492803ms”,“path\_left”:“/var/lib/weaviate/Book/4921/lsm/property\_exploit\_available\_nullState/segment-1731491661482528037.db”,“path\_right”:“/var/lib/weaviate/Book/4921/lsm/property\_exploit\_available\_nullState/segment-1731491804128129752.db”,“segment\_index”:7,“shard”:“4921”,“time”:“2024-11-13T09:59:06Z”,“took”:343492803}

Hey @DudaNogueira  
In this logs what is “took”:343492803. What is problem with weaviate ?

---

<div class="post-metadata">

**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\
**Post date:** [November 13, 2024, 12:08pm UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/8 "2024-11-13T12:08:58Z")

</div>

**I have multiple cases and need help optimizing the code:**

1. **Case 1** : Suppose I have an additional property, `"is_new"`, in the Book schema. Initially, `is_new` will be set to `True` when a book is added for the first time. If the same book title appears again, `is_new` should be set to `False`.

```auto
wcc.Property(
            name="is_new",
            data_type=wcc.DataType.BOOL         
        )

```

**Case 2** : I have another collection called `Publication`, which has its own properties and a cross-reference between `Book` and `Publication`.

```auto
 properties=[
        wcc.Property(
            name="Publication_Title",
            data_type=wcc.DataType.TEXT,
        ),
        wcc.Property(
            name="Publication_Date",
            data_type=wcc.DataType.TEXT,
             ),
        wcc.ReferenceProperty(
            name="Book_Reference",
            target_collection="Book,
    ],

```

I have two separate files for ingestion: book.csv, which contains author details, and publication.csv, which contains publication details. Each collection is ingested separately, with book.csv ingested first, followed by publication.csv.  
During `publication.csv` ingestion, we need to check if each book exists in the Book collection (mapping by a non-UUID field), fetch its UUID, and then link it to Publication.  
How can we verify ( if Book exist ) then link ( Publication & Book), and ingest the data effectively?

---

<div class="post-metadata">

**Author:** ![Shahin](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/shahin/32/7944_2.png) [@Shahin](https://forum.weaviate.io/u/Shahin)\
**Post date:** [November 15, 2024, 10:45am UTC](https://forum.weaviate.io/t/i-am-getting-hnsw-vector-cache-prefill-frequently/7501/9 "2024-11-15T10:45:58Z")

</div>

Hey @2020ashish, as per thread [Metadata properties - #3 by 2020ashish](https://forum.weaviate.io/t/metadata-properties/7572/3),

My answer based on your last message in this thread, since you have two files and are batching each separately, I would recommend going with a cross-reference approach. It will work well in your case, especially when you need to query and filter metadata. I wouldn’t go for a boolean property (is\_new) unless there’s a compelling reason for it.

With two collections and one cross-reference, performance should not be an issue unless your queries become unusually complex. Based on my experience with similar use cases, a cross-reference is a solution for this scenario. This approach is clean, and maintainable for batching workflow.

Does that help?