# Hnsw load commit log corruption error **URL:** \ **Category:** Support\ **Created:** [February 13, 2024, 1:04pm UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470 "2024-02-13T13:04:54Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\ **Post date:** [February 13, 2024, 1:04pm UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/1 "2024-02-13T13:04:54Z")
### Description Weaviate got crash… Log showed this… I’m completely clue less write ahead log ended abruptly ### Server Setup Information - Weaviate Server Version: 1.23.7 - Deployment Method: - Multi Node? Number of Running Nodes: - Client Language and Version: V4 ![image](https://canada1.discourse-cdn.com/flex027/uploads/weaviate/original/1X/1bcef42a381c28bf0782f623a5f5e8e1bfdf139f.png) ### Any additional Information ---
**Author:** ![Grzegorz\_Pasieka](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/grzegorz_pasieka/32/974_2.png) [@Grzegorz\_Pasieka](https://forum.weaviate.io/u/Grzegorz_Pasieka)\ **Post date:** [February 14, 2024, 5:32pm UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/2 "2024-02-14T17:32:23Z")
Hi, we’ve got the same error since yesterday ```auto {"action":"hnsw_load_commit_log_corruption","level":"error","msg":"write-ahead-log ended abruptly, some elements may not have been recovered","path":"/var/lib/weaviate/aiskillv2/1efo8r43N3tC/main.hnsw.commitlog.d/1707929440","time":"2024-02-14T17:00:55Z"} {"level":"info","msg":"Completed loading shard aiskillv2_1efo8r43N3tC in 55.950096536s","time":"2024-02-14T17:00:55Z"} {"level":"info","msg":"Completed loading shard aiskillv2_5SQtIKAXTNQ6 in 56.630300501s","time":"2024-02-14T17:01:52Z"} {"action":"requests_total","api":"graphql","class_name":"","error":"context canceled","level":"error","msg":"unexpected error","query_type":"","time":"2024-02-14T17:03:36Z"} {"description":"An I/O timeout occurs when the request takes longer than the specified server-side timeout.","error":"write tcp 10.9.22.24:8080-\u003e127.0.0.6:39361: i/o timeout","hint":"Either try increasing the server-side timeout using e.g. '--write-timeout=600s' as a command line flag when starting Weaviate, or try sending a computationally cheaper request, for example by reducing a batch size, reducing a limit, using less complex filters, etc. Note that this error is only thrown if client-side and server-side timeouts are not in sync, more precisely if the client-side timeout is longer than the server side timeout.","level":"error","method":"POST","msg":"i/o timeout","path":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/graphql","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"time":"2024-02-14T17:04:48Z"} {"action":"requests_total","api":"graphql","class_name":"","error":"context canceled","level":"error","msg":"unexpected error","query_type":"","time":"2024-02-14T17:06:20Z"} {"action":"requests_total","api":"graphql","class_name":"","error":"context canceled","level":"error","msg":"unexpected error","query_type":"","time":"2024-02-14T17:07:48Z"} {"description":"An I/O timeout occurs when the request takes longer than the specified server-side timeout.","error":"write tcp 10.9.22.24:8080-\u003e127.0.0.6:46245: i/o timeout","hint":"Either try increasing the server-side timeout using e.g. '--write-timeout=600s' as a command line flag when starting Weaviate, or try sending a computationally cheaper request, for example by reducing a batch size, reducing a limit, using less complex filters, etc. Note that this error is only thrown if client-side and server-side timeouts are not in sync, more precisely if the client-side timeout is longer than the server side timeout.","level":"error","method":"POST","msg":"i/o timeout","path":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/graphql","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"time":"2024-02-14T17:08:31Z"} {"description":"An I/O timeout occurs when the request takes longer than the specified server-side timeout.","error":"write tcp 10.9.22.24:8080-\u003e127.0.0.6:36071: i/o timeout","hint":"Either try increasing the server-side timeout using e.g. '--write-timeout=600s' as a command line flag when starting Weaviate, or try sending a computationally cheaper request, for example by reducing a batch size, reducing a limit, using less complex filters, etc. Note that this error is only thrown if client-side and server-side timeouts are not in sync, more precisely if the client-side timeout is longer than the server side timeout.","level":"error","method":"POST","msg":"i/o timeout","path":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/graphql","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"time":"2024-02-14T17:13:40Z"} {"action":"requests_total","api":"graphql","class_name":"","error":"context canceled","level":"error","msg":"unexpected error","query_type":"","time":"2024-02-14T17:13:43Z"} {"description":"An I/O timeout occurs when the request takes longer than the specified server-side timeout.","error":"write tcp 10.9.22.24:8080-\u003e127.0.0.6:34179: i/o timeout","hint":"Either try increasing the server-side timeout using e.g. '--write-timeout=600s' as a command line flag when starting Weaviate, or try sending a computationally cheaper request, for example by reducing a batch size, reducing a limit, using less complex filters, etc. Note that this error is only thrown if client-side and server-side timeouts are not in sync, more precisely if the client-side timeout is longer than the server side timeout.","level":"error","method":"POST","msg":"i/o timeout","path":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/graphql","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"time":"2024-02-14T17:14:17Z"} {"description":"An I/O timeout occurs when the request takes longer than the specified server-side timeout.","error":"write tcp 10.9.22.24:8080-\u003e127.0.0.6:38333: i/o timeout","hint":"Either try increasing the server-side timeout using e.g. '--write-timeout=600s' as a command line flag when starting Weaviate, or try sending a computationally cheaper request, for example by reducing a batch size, reducing a limit, using less complex filters, etc. Note that this error is only thrown if client-side and server-side timeouts are not in sync, more precisely if the client-side timeout is longer than the server side timeout.","level":"error","method":"POST","msg":"i/o timeout","path":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/graphql","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"time":"2024-02-14T17:25:32Z"} {"level":"error","msg":" memberlist: Failed fallback TCP ping: timeout 1s: read tcp 10.9.22.24:44408-\u003e10.9.23.28:7000: i/o timeout","time":"2024-02-14T17:30:48Z"} {"level":"info","msg":" memberlist: Suspect weaviate-0 has failed, no acks received","time":"2024-02-14T17:30:48Z"} {"level":"error","msg":" memberlist: Failed fallback TCP ping: timeout 1s: read tcp 10.9.22.24:44428-\u003e10.9.23.28:7000: i/o timeout","time":"2024-02-14T17:30:50Z"} {"level":"info","msg":" memberlist: Suspect weaviate-0 has failed, no acks received","time":"2024-02-14T17:30:50Z"} {"level":"info","msg":" memberlist: Marking weaviate-0 as failed, suspect timeout reached (0 peer confirmations)","time":"2024-02-14T17:30:52Z"} {"level":"error","msg":" memberlist: Failed fallback TCP ping: timeout 1s: read tcp 10.9.22.24:45276-\u003e10.9.23.28:7000: i/o timeout","time":"2024-02-14T17:30:52Z"} {"level":"info","msg":" memberlist: Suspect weaviate-0 has failed, no acks received","time":"2024-02-14T17:30:52Z"} ``` ---
**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\ **Post date:** [February 14, 2024, 11:40pm UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/3 "2024-02-14T23:40:18Z")
Hi! Welcomet to our community, @Grzegorz_Pasieka !! 🤗 What are the versions you are running? I believe this is a recent bug our team was recently able to sort out. The ones that was really hard to reproduce. If that is the same, it will only happen in very specific circunstances (crash, restart and crash/restart in a specifc start cycle) with some delete in the mix. There are some PRs already. Follow this issue for for more on related issues that is being worked on this after this discovery: > > > \### How to reproduce this bug? > > Not known yet - working on reproduction. Few ind…ependent sightings > > \### What is the expected behavior? > > BM25 index is crash-resistant > > \### What is the actual behavior? > > Users occasionally see panics like the following: > > \`\`\` > panic: runtime error: index out of range \[16\] with length 16 > > goroutine 50113 \[running\]: > github.com/weaviate/weaviate/adapters/repos/db/inverted.(\*BM25Searcher).createTerm(0xc09dd078c0, 0x413b3f1f00000000, {0x0, 0x0}, {0xc002f08edd, 0xe}, {0xc002a2ef00, 0xe, 0x10?}, 0xc03bb94fc0?, ...) > /go/src/github.com/weaviate/weaviate/adapters/repos/db/inverted/bm25\_searcher.go:431 +0x126e > github.com/weaviate/weaviate/adapters/repos/db/inverted.(\*BM25Searcher).wand.func1() > \`\`\` > > \### Supporting information > > We are aware and investigating. If you have a reliable way to reproduce this (from scratch), please do let us know. > > \### Server Version > > Spotted on 1.22.x and 1.23.x > > \### Code of Conduct > > \- \[X\] I have read and agree to the Weaviate's \[Contributor Guide\](https://weaviate.io/developers/contributor-guide) and \[Code of Conduct\](https://weaviate.io/service/code-of-conduct) Thanks! ---
**Author:** ![Grzegorz\_Pasieka](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/grzegorz_pasieka/32/974_2.png) [@Grzegorz\_Pasieka](https://forum.weaviate.io/u/Grzegorz_Pasieka)\ **Post date:** [February 15, 2024, 12:34am UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/4 "2024-02-15T00:34:10Z")
weviate: v1.23.3 helm-chart: v16.8.0 Thanks for your response! ---
**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\ **Post date:** [February 15, 2024, 2:20am UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/5 "2024-02-15T02:20:03Z")
weviate: v1.23.3 helm-chart: v16.8.0 Multi Node : Single node Client Language : Python & version : ‘4.4.4’ ---
**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\ **Post date:** [February 15, 2024, 1:13pm UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/6 "2024-02-15T13:13:09Z")
Ok. A patch to fix this is being developed. Is your server in a crash loop? There is one “variant” of this bug that upgrading will keep it from crashlooping for now. Not sure this is the same situation, but it’s always good to run latest versions. Here: > **[Release v1.23.7 - BM25/WAND internal errors logic improvement, gRPC API...](https://github.com/weaviate/weaviate/releases/tag/v1.23.7)** > > Breaking Changes > none > New Features > none > Fixes > > Send uuid as bytes for groupby operations by @dirkkul in #4090 > Handle internal errors in BM25/WAND logic more gracefully by @etiennedi in #4100 > Add te... We’ll keep you posted here for any news. Thanks! ---
**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\ **Post date:** [March 13, 2024, 5:41am UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/7 "2024-03-13T05:41:39Z")
weviate: v1.23.7 helm-chart: v16.8.0 Multi Node : Single node Client Language : Python & version : 3.25.3 We are getting error again {“action”:“hnsw\_load\_commit\_log\_corruption”,“level”:“error”,“msg”:“write-ahead-log ended abruptly, some elements may not have been recovered”,“path”:“/var/lib/weaviate/Name/5822/main.hnsw.commitlog.d/1710143126”,“time”:“2024-03-13T04:43:39Z”} {“action”:“hnsw\_load\_commit\_log\_corruption”,“level”:“error”,“msg”:“write-ahead-log ended abruptly, some elements may not have been recovered”,“path”:“/var/lib/weaviate/assets\_info/5822/main.hnsw.commitlog.d/1710145731”,“time”:“2024-03-13T04:43:41Z”} What is actual problem? can u please take a look into it. When we are ingesting data, we always getting connection error after some time… Currently testing weavaite on config: CPU limit - 2000m Memory limit- 7.81GiB with multi tenant ---
**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\ **Post date:** [March 13, 2024, 9:20pm UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/8 "2024-03-13T21:20:02Z")
Hi! Was there any out of memory or out of hard disk event? We strongly suggest using python client v4, and the dynamic batch size for ingesting a lot of data in a short period of time. The dynamic method will listen for the server feedback and adjust the batch size accordingly, prevent overloading the server. Do you have any reading of memory usage when this event happened? Thanks! ---
**Author:** ![2020ashish](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/2020ashish/32/797_2.png) [@2020ashish](https://forum.weaviate.io/u/2020ashish)\ **Post date:** [March 14, 2024, 4:39am UTC](https://forum.weaviate.io/t/hnsw-load-commit-log-corruption-error/1470/9 "2024-03-14T04:39:31Z")
There is not a single out of memory event. Here is grafana SS for past 24 Hr ![image](https://canada1.discourse-cdn.com/flex027/uploads/weaviate/original/2X/c/c61adce5d581e1184055b7b9cd6c91a836008460.png)