Error when Data Stored in AWS EFS

I have created a Weaviate Setup in AWS ECS and create auto-scaling. Due to spike load, the weaviate instance restarted and now, I am getting below error which accessing the data.

{
“data”: {
“Get”: {
“WeaviateDemo3”: null
}
},
“errors”: [
{
“locations”: [
{
“column”: 6,
“line”: 1
}
],
“message”: “explorer: list class: search: object search at index weaviatedemo3: local shard object search weaviatedemo3_PXksOv6syq7H: Unable to load shard PXksOv6syq7H: init shard "weaviatedemo3_PXksOv6syq7H": init shard "weaviatedemo3_PXksOv6syq7H": shard db: create objects bucket: init disk segments: init segment segment-1719829104648996364.db: mmap file: invalid argument”,
“path”: [
“Get”,
“WeaviateDemo3”
]
}
]
}

@DudaNogueira @jphwang

Hi!

Do you have any logs from the server side?

Note that, for Weaviate, as long as there is writable disk space, it should work properly.

I found below in logs. Please let me know if this helps.

{"action":"lsm_memtable_flush","class":"WeaviateDemo3","error":"flush: unlinkat /var/lib/weaviate/weaviatedemo3/PXksOv6syq7H/lsm/objects/segment-1719829104648996364.scratch.d: directory not empty","index":"weaviatedemo3","level":"error","msg":"flush and switch failed","path":"/var/lib/weaviate/weaviatedemo3/PXksOv6syq7H/lsm/objects","shard":"PXksOv6syq7H","time":"2024-07-01T10:18:50Z"}

@DudaNogueira

Seems that it raised an error due to this folder not being empty :thinking:

PLease, when opening technical support thread, select the Support category and answer this:

  • Weaviate Server Version:
  • Deployment Method:
  • Multi Node? Number of Running Nodes:
  • Client Language and Version:
  • Multitenancy?:

Mainly: what version are you runnig?

for reference, this error traces back to:

Thanks for your response.

1 Like