# Empty search results when one node (of 3) is not fully replicated

**URL:** <https://forum.weaviate.io/t/empty-search-results-when-one-node-of-3-is-not-fully-replicated/3971>\
**Category:** Support\
**Tags:** technical\
**Created:** [September 3, 2024, 8:43pm UTC](https://forum.weaviate.io/t/empty-search-results-when-one-node-of-3-is-not-fully-replicated/3971 "2024-09-03T20:43:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![wvuser](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/wvuser/32/2371_2.png) [@wvuser](https://forum.weaviate.io/u/wvuser)\
**Post date:** [September 3, 2024, 8:43pm UTC](https://forum.weaviate.io/t/empty-search-results-when-one-node-of-3-is-not-fully-replicated/3971/1 "2024-09-03T20:43:10Z")

</div>

### Description

We are testing 3-node cluster (replication factor 3), all data synchronized and… suddenly one node physically failed. The new pod (empty) was deployed on a new machine and activated “read repair” procedure (using batch reading), but It work too slowly ~50obj/sec (_how to speedup it?_).  
**Problem:** when search request (text/vector) is routed to “empty” node (with any QUORUM or ALL consistency level) - not valid result avalaible. It seems that the “empty” node explicitly looks for data within itself at first… not founds them and quorum/all conditions are not satisfied. At result:

- for “property equal” search (getting objects by id’s list at once with “ContainsAny”) - not any results returned;
- for vector search - not self and not closest vectors returned.

If request routed to any node “with this data”, search results are correct.  
How to exclude “empty” (not replicated) node from operations or QUORUM/ALL conditions?

### Server Setup Information

- Weaviate Server Version: 1.25.12
- Deployment Method: k8s
- Multi Node? Number of Running Nodes: Yes, 3
- Client Language and Version: Python3, Python Client v3
- Multitenancy?: No

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [September 4, 2024, 1:30pm UTC](https://forum.weaviate.io/t/empty-search-results-when-one-node-of-3-is-not-fully-replicated/3971/2 "2024-09-04T13:30:09Z")

</div>

hi @wvuser !!

Are you aware of the new feature that landed in 1.26, the [async replication](https://weaviate.io/developers/weaviate/concepts/replication-architecture/consistency#async-replication)?

> **[Replication | Weaviate](https://weaviate.io/developers/weaviate/configuration/replication#data-consistency)**
>
> Weaviate instances can be replicated. Replication can improve read throughput, improve availability, and enable zero-downtime upgrades.

This will proactively start the repair async.

It may reduce the time between the node realizing that it is inconsistent and a part of a cluster.

I will ask about this scenario to our team.

Thanks!

---

<div class="post-metadata">

**Author:** ![wvuser](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/wvuser/32/2371_2.png) [@wvuser](https://forum.weaviate.io/u/wvuser)\
**Post date:** [September 6, 2024, 6:19am UTC](https://forum.weaviate.io/t/empty-search-results-when-one-node-of-3-is-not-fully-replicated/3971/3 "2024-09-06T06:19:32Z")

</div>

> [@DudaNogueira](#):
>
> Are you aware of the new feature that landed in 1.26, the [async replication](https://weaviate.io/developers/weaviate/concepts/replication-architecture/consistency#async-replication)?

Yes:) Tried to upgrade:

1. Pods succesfully started with 1.26.3 image version;
2. Manually updated schema with **“asyncEnabled”:true** ;
3. In logs appeared information about async replication starand crash:  
_{“action”:“async\_replication”,“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“class\_name”:“MyClass”,“level”:“info”,“msg”:“hashtree initialization is progress…”,“object\_count”:14842232,“shard\_name”:“QZOXu2zY6rKM”,“time”:“2024-09-05T09:49:23Z”}_  
_{“action”:“async\_replication”,“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“class\_name”:“MyClass”,“level”:“info”,“msg”:“hashbeater stopped”,“shard\_name”:“b4ulnsitcsN2”,“time”:“2024-09-05T09:49:24Z”}_  
_{“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“level”:“error”,“msg”:“Recovered from panic: runtime error: index out of range [1] with length 1”,“time”:“2024-09-05T09:49:24Z”}_  
_goroutine 692758 [running]:_  
_runtime/debug.Stack()_

- /usr/local/go/src/runtime/debug/stack.go:24 +0x5e\*  
_runtime/debug.PrintStack()_
- /usr/local/go/src/runtime/debug/stack.go:16 +0x13\*  
_[github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1()](http://github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1())_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:32 +0x150\*  
_panic({0x20565e0?, 0xf71f95d218?})_
- /usr/local/go/src/runtime/panic.go:914 +0x21f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_segment).newCursorWithSecondaryIndex(0xc002ae0870, 0x1)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_segment\_replace.go:58 +0x290\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_SegmentGroup).newCursorsWithSecondaryIndex(0xc0052dc0c0, 0xf4def49c70?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_segment\_replace.go:99 +0xab\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_Bucket).CursorWithSecondaryIndex(0xc003e64000, 0x21996ef?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_bucket\_replace.go:82 +0x9c\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).ObjectDigestsByTokenRange(0xc0050ff180?, {0x29050a0?, 0xc029e42780?}, 0x150000000000000, 0x19dffffffffffff, 0x3e8)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_read.go:138 +0x9f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Index).DigestObjectsInTokenRange(0x459eab?, {0x29050a0, 0xc029e42780}, {0xc005823a90, 0xc}, 0xf4def4a0d8?, 0x565ff2?, 0x0?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/replication.go:475 +0x169\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).stepsTowardsShardConsistency(0xc005466000, {0x29050a0, 0xc029e42780}, {0xc005823a90, 0xc}, {0xc5f4305e78, 0x11}, 0xc5f434d440?, 0x19dffffffffffff, 0x186a0)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:336 +0x105\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).hashBeat(0xc005466000)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:288 +0x81f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).initHashBeater.func1()_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:80 +0x45b\*  
_[github.com/weaviate/weaviate/entities/errors.GoWrapper.func1()](http://github.com/weaviate/weaviate/entities/errors.GoWrapper.func1())_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:36 +0x62\*  
_created by [github.com/weaviate/weaviate/entities/errors.GoWrapper](http://github.com/weaviate/weaviate/entities/errors.GoWrapper) in goroutine 580955_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:26 +0x79\*  
_{“action”:“async\_replication”,“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“class\_name”:“MyClass”,“level”:“info”,“msg”:“hashbeater stopped”,“shard\_name”:“PfYchyFAyseL”,“time”:“2024-09-05T09:49:24Z”}_  
_{“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“level”:“error”,“msg”:“Recovered from panic: runtime error: index out of range [1] with length 1”,“time”:“2024-09-05T09:49:24Z”}_  
_goroutine 662155 [running]:_  
_runtime/debug.Stack()_
- /usr/local/go/src/runtime/debug/stack.go:24 +0x5e\*  
_runtime/debug.PrintStack()_
- /usr/local/go/src/runtime/debug/stack.go:16 +0x13\*  
_[github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1()](http://github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1())_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:32 +0x150\*  
_panic({0x20565e0?, 0xf737c47218?})_
- /usr/local/go/src/runtime/panic.go:914 +0x21f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_segment).newCursorWithSecondaryIndex(0xc002b4e000, 0x1)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_segment\_replace.go:58 +0x290\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_SegmentGroup).newCursorsWithSecondaryIndex(0xc00553a300, 0xf6e26e1c70?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_segment\_replace.go:99 +0xab\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_Bucket).CursorWithSecondaryIndex(0xc004034000, 0x21996ef?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_bucket\_replace.go:82 +0x9c\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).ObjectDigestsByTokenRange(0xc0050ff180?, {0x29050a0?, 0xc029e428c0?}, 0x2fe000000000000, 0x4b3ffffffffffff, 0x3e8)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_read.go:138 +0x9f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Index).DigestObjectsInTokenRange(0x459eab?, {0x29050a0, 0xc029e428c0}, {0xc0058229f0, 0xc}, 0xf6e26e20d8?, 0x565ff2?, 0x0?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/replication.go:475 +0x169\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).stepsTowardsShardConsistency(0xc005a20000, {0x29050a0, 0xc029e428c0}, {0xc0058229f0, 0xc}, {0xe825b2cb88, 0x11}, 0xf6e26b0300?, 0x4b3ffffffffffff, 0x186a0)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:336 +0x105\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).hashBeat(0xc005a20000)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:288 +0x81f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).initHashBeater.func1()_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:80 +0x45b\*  
_[github.com/weaviate/weaviate/entities/errors.GoWrapper.func1()](http://github.com/weaviate/weaviate/entities/errors.GoWrapper.func1())_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:36 +0x62\*  
_created by [github.com/weaviate/weaviate/entities/errors.GoWrapper](http://github.com/weaviate/weaviate/entities/errors.GoWrapper) in goroutine 580957_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:26 +0x79\*  
_{“action”:“async\_replication”,“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“class\_name”:“MyClass”,“level”:“info”,“msg”:“hashtree initialization is progress…”,“object\_count”:15064448,“shard\_name”:“QZOXu2zY6rKM”,“time”:“2024-09-05T09:49:24Z”}_  
_{“action”:“async\_replication”,“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“class\_name”:“MyClass”,“level”:“info”,“msg”:“hashtree successfully initialized”,“shard\_name”:“QZOXu2zY6rKM”,“time”:“2024-09-05T09:49:24Z”}_  
_{“action”:“async\_replication”,“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“class\_name”:“MyClass”,“level”:“info”,“msg”:“hashbeater started…”,“shard\_name”:“QZOXu2zY6rKM”,“time”:“2024-09-05T09:49:24Z”}_  
_{“action”:“async\_replication”,“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“class\_name”:“MyClass”,“level”:“info”,“msg”:“hashbeater stopped”,“shard\_name”:“QZOXu2zY6rKM”,“time”:“2024-09-05T09:49:28Z”}_  
_{“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“level”:“error”,“msg”:“Recovered from panic: runtime error: index out of range [1] with length 1”,“time”:“2024-09-05T09:49:28Z”}_  
_goroutine 695187 [running]:_  
_runtime/debug.Stack()_
- /usr/local/go/src/runtime/debug/stack.go:24 +0x5e\*  
_runtime/debug.PrintStack()_
- /usr/local/go/src/runtime/debug/stack.go:16 +0x13\*  
_[github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1()](http://github.com/weaviate/weaviate/entities/errors.GoWrapper.func1.1())_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:32 +0x150\*  
_panic({0x20565e0?, 0xf653e906a8?})_
- /usr/local/go/src/runtime/panic.go:914 +0x21f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_segment).newCursorWithSecondaryIndex(0xc003f803c0, 0x1)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_segment\_replace.go:58 +0x290\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_SegmentGroup).newCursorsWithSecondaryIndex(0xc0040bca80, 0xf4def49c70?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_segment\_replace.go:99 +0xab\*  
\*[github.com/weaviate/weaviate/adapters/repos/db/lsmkv.](http://github.com/weaviate/weaviate/adapters/repos/db/lsmkv.)(_Bucket).CursorWithSecondaryIndex(0xc005a2e240, 0x21996ef?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/cursor\_bucket\_replace.go:82 +0x9c\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).ObjectDigestsByTokenRange(0xc0050ff180?, {0x29050a0?, 0xc029e42820?}, 0x0, 0x1ffffffffffff, 0x3e8)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_read.go:138 +0x9f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Index).DigestObjectsInTokenRange(0x459eab?, {0x29050a0, 0xc029e42820}, {0xc005823240, 0xc}, 0xf4def4a0d8?, 0x565ff2?, 0x0?)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/replication.go:475 +0x169\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).stepsTowardsShardConsistency(0xc0004b1dc0, {0x29050a0, 0xc029e42820}, {0xc005823240, 0xc}, {0xf737c47578, 0x11}, 0xf68ad82a40?, 0x1ffffffffffff, 0x186a0)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:336 +0x105\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).hashBeat(0xc0004b1dc0)_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:288 +0x81f\*  
\*[github.com/weaviate/weaviate/adapters/repos/db.](http://github.com/weaviate/weaviate/adapters/repos/db.)(_Shard).initHashBeater.func1()_
- /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard\_hashbeater.go:80 +0x45b\*  
_[github.com/weaviate/weaviate/entities/errors.GoWrapper.func1()](http://github.com/weaviate/weaviate/entities/errors.GoWrapper.func1())_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:36 +0x62\*  
_created by [github.com/weaviate/weaviate/entities/errors.GoWrapper](http://github.com/weaviate/weaviate/entities/errors.GoWrapper) in goroutine 580956_
- /go/src/github.com/weaviate/weaviate/entities/errors/go\_wrapper.go:26 +0x79\*  
_{“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“level”:“debug”,“msg”:" memberlist: Initiating push/pull sync with: weaviate-2 xxx.yyy.zzz.www:7000",“time”:“2024-09-05T09:49:31Z”}_  
_{“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“level”:“debug”,“msg”:" memberlist: Stream connection from=xxx.yyy.zzz.www:53136",“time”:“2024-09-05T09:49:33Z”}_  
_{“build\_git\_commit”:“9a4ea6d”,“build\_go\_version”:“go1.21.13”,“build\_image\_tag”:“1.26.3”,“build\_wv\_version”:“1.26.3”,“level”:“debug”,“msg”:" memberlist: Stream connection from=xxx.yyy.zzz.www:33106",“time”:“2024-09-05T09:49:41Z”}_

1. Async replication didn’t starts any more;
2. Tried to restart pods (too long termination time! after ‘timeout’ pod terminated forcelly);
3. After load **“asyncEnabled”** flag auto resetted to **false**.

> [@wvuser](#):
>
> It seems that the “empty” node explicitly looks for data within itself at first… not founds them and quorum ~~/all~~ conditions are not satisfied.

Problem with “empty for text (or not closest for vectors)” search results with **QUORUM** consistency when requests routed to “empty” node are stay…