Nodes/shards objects imbalance

[wvuser]

Description

We have a problem with replica synchronization (deletions not fixed when nodes fail/restart) in a cluster of 3 nodes and 3 replicas. Running batch/cursor read process with ConsistencyLevel=ALL does not help. Tried even full reading with connect (route) to different (every) nodes.

Some cluster info, private fields are masked (ip-addresses, schema/property names):

• Cluster statistics
• Nodes info
• Node #1 file-list
• Node #2 file-list
• Node #3 file-list

Found some issues:

• Support Read-Repairs (and async repair) for deleted objects
• Object count differs

Are any advices avalaible?

Can you tell about ‘SEARCH_FORCE_EXPAND_TO_ALL_REPLICAS’ flag:
Add flag to expand search to all available replicas
How to use it? Not found any use in v1.25.12 sources (or I’m wrong?).

Server Setup Information

• Weaviate Server Version: 1.25.12
• Deployment Method: k8s
• Multi Node? Number of Running Nodes: Yes, 3
• Client Language and Version: Python3, Python Client v3
• Multitenancy?: No