Pod metric issue

I have inserted some mock data into my k3s weaviate cluster using this:
import weaviate

import weaviate.classes.config as wc

NLB_DNS_NAME = os.getenv(“AWS_NLB”)

client = weaviate.connect_to_custom(

http_host=NLB_DNS_NAME,

http_port=8080,  

http_secure=False,

grpc_host=NLB_DNS_NAME,

grpc_port=50051,  

grpc_secure=False

)

try:

 if client.is_ready():

    print("Successfully connected to Weaviate.")

else:

    print("Failed to connect to Weaviate.")

    client.close()

    exit(1)

client.collections.create(

    name="Exchange",

    properties=\[

        wc.Property(name="name", data_type=wc.DataType.TEXT),

        wc.Property(name="location", data_type=wc.DataType.TEXT),

        wc.Property(name="founded_year", data_type=wc.DataType.NUMBER),

    \],

    sharding_config=wc.Configure.sharding(desired_count=3),

    replication_config=wc.Configure.replication(factor=2),

    \# Optional: Add a vectorizer if needed; here using none for simplicity

    vectorizer_config=wc.Configure.Vectorizer.none()

)




exchange_collection = client.collections.get("Exchange")



\# Mock data for stock exchanges

mock_data = \[

    {"name": "New York Stock Exchange", "location": "New York, USA", "founded_year": 1792},

    {"name": "NASDAQ", "location": "New York, USA", "founded_year": 1971},

    {"name": "London Stock Exchange", "location": "London, UK", "founded_year": 1801},

    {"name": "Tokyo Stock Exchange", "location": "Tokyo, Japan", "founded_year": 1878}

\]




with exchange_collection.batch.dynamic() as batch:

    for data in mock_data:

        batch.add_object(

            properties=data

        )

Now the second code:

result = exchange_collection.query.fetch_objects(limit=10)

print(f"Total objects in collection: {len(result.objects)}")



for obj in result.objects:

    print(f"- {obj.properties\['name'\]} ({obj.properties\['location'\]}) - Founded: {obj.properties\['founded_year'\]}")

finally:

client.close()

With this code im getting the following as output:
Total objects found: 4

  • London Stock Exchange (UUID: 60d6c2fe-07ef-4928-8aba-…..)

  • New York Stock Exchange (UUID: 6d439d55-1651-4544-…..)

  • Tokyo Stock Exchange (UUID: b3e30b81-eda5-4e90-ab53-…..)

  • NASDAQ (UUID: bcacee6a-f83d-4dac-b6d7-…..)

    but with this code im not able to see the object inside shards:
    3rd code:

    NLB = os.getenv(“AWS_NLB”)

    COLLECTION = “Exchange” # collection you want to inspect

    client = weaviate.connect_to_custom(

    http_host=NLB,  http_port=8080,  http_secure=False,
    
    grpc_host=NLB,  grpc_port=50051, grpc_secure=False,
    

    )

    def print_distribution(collection: str = COLLECTION) → None:

    """
    
    Fetch verbose cluster info once and print object distribution.
    
    """
    
    nodes = client.cluster.nodes(output="verbose")   # one round-trip
    
    
    
    print(f"\\nLive object distribution for collection '{collection}':\\n")
    
    
    
    for node in nodes:
    
        print(f"Pod: {node.name}   Status: {node.status}")
    
        shards_for_col = \[s for s in node.shards if s.collection == collection\]
    
    
    
        if not shards_for_col:
    
            print("  • No shards for this collection on this pod\\n")
    
            continue
    
    
    
        for shard in shards_for_col:
    
            print(f"  • Shard {shard.name}  objects={shard.object_count}")
    
        print()
    

    Pod: weaviate-0 Status: HEALTHY
    • Shard hokJEudIj… objects=0
    • Shard YVmYEwFNC… objects=0

    Pod: weaviate-1 Status: HEALTHY
    • Shard cQMk2AvqW… objects=0
    • Shard hokJEudIj… objects=0

    Pod: weaviate-2 Status: HEALTHY
    • Shard YVmYEwFNC… objects=0
    • Shard cQMk2AvqW… objects=0

Is there any bug in weaviate ?

Hi @Saketh !!

Thanks for reporting!!!

I was able to replicate this on 1.32.3, and not on 1.30.1 (random version I tried).

I will investigate this further, but it indeed smells like a bug :grimacing:

Hi @DudaNogueira , after ingesting more data we were able to see the distribution of objects across different pods in our k3s weaviate cluster.

Hi @DudaNogueira , after ingesting more data we were able to see the distribution of objects across different pods in our k3s weaviate cluster

.

hi @Saketh !

Thanks for the feedback. I have check with our core team, and that metric is eventually consistent, and it was due to this change: https://github.com/weaviate/weaviate/pull/4203

So in order to get the total count, it is best to indeed aggregate.

Thanks!