Embedded weaviate

Support
1

Description

I’m using embedded weaviate since three month now and i never face major issue.
And toaday when i try to launch my code it raise an memory error and set the mode to READONLY
My code create chunk from a document and store them into a collection if the collection already exist it is deleted in order to create a new one.

Server Setup Information

self.client = weaviate.connect_to_embedded(
            version="1.24.1",
        )

What I already tried

  • Delete all collection but it is not working
  • Go into my persistent data storage and delete all file from there. It look like it is working but not always

Here are the logs i get from the server

INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/Rag_collection "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: DELETE http://localhost:8079/v1/schema/Rag_collection "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET http://localhost:8079/v1/schema/Rag_collection "HTTP/1.1 404 Not Found"
{"level":"info","msg":"Created shard rag_collection_OcEOzOgZ2vt1 in 476.834µs","time":"2024-06-25T14:09:24+02:00"}
{"action":"hnsw_vector_cache_prefill","count":1000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-06-25T14:09:24+02:00","took":53601}
INFO:httpx:HTTP Request: POST http://localhost:8079/v1/schema "HTTP/1.1 200 OK"
{"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:dce323e3-1390-4843-921a-82a74666bdae Type:INIT Version:1.24.1 Modules:generative-openai,qna-openai,ref2vec-centroid,reranker-cohere,text2vec-cohere,text2vec-huggingface,text2vec-openai NumObjects:0 OS:linux Arch:amd64}","time":"2024-06-25T14:09:24+02:00"}
{"action":"read_disk_use","level":"warning","msg":"disk usage currently at 92.09%, threshold set to 80.00%","path":"/home/loic/.local/share/weaviate","time":"2024-06-25T14:09:24+02:00"}
{"level":"info","msg":"Completed loading shard neovision_collection_mjdBJOsYHcr5 in 822.755µs","time":"2024-06-25T14:09:24+02:00"}
{"action":"lsm_compaction","class":"Neovision_collection","index":"neovision_collection","level":"warning","msg":"compaction halted due to shard READONLY status","path":"/home/loic/.local/share/weaviate/neovision_collection/mjdBJOsYHcr5/lsm","shard":"mjdBJOsYHcr5","time":"2024-06-25T14:09:24+02:00"}
{"action":"lsm_compaction","class":"Rag_collection","index":"rag_collection","level":"warning","msg":"compaction halted due to shard READONLY status","path":"/home/loic/.local/share/weaviate/rag_collection/OcEOzOgZ2vt1/lsm","shard":"OcEOzOgZ2vt1","time":"2024-06-25T14:09:24+02:00"}
{"action":"set_shard_read_only","level":"warning","msg":"Set READONLY, disk usage currently at 92.09%, threshold set to 90.00%","path":"/home/loic/.local/share/weaviate","time":"2024-06-25T14:09:24+02:00"}
{"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-06-25T14:09:24+02:00","took":40239}

Questions

  • How to delete those file that take too much space ?
  • To what correspond the disk storage cause i still much memory on my disk ?

Thanks in advance

2

hi @engelsl !!

As you already know, the embedded is experimental, and shouldn’t be used in production :see_no_evil:

Have you tried setting your shards back to READY?

When Weaviate reaches a certain threshold of available disk, it will enter READONLY mode to avoid file corruption, as described here:

Let me know if this helps.

Thanks!

3

Hi @DudaNogueira

You are right embedded is still experimental and i use it on a R&D project.

Documents you share helps me to understand what happen thanks a lot

Maybe it was not clear on my previous message, my issue was how to clear the disk from all it’s data