Replication config in class schema

Support
1

Description

Following code works only when I comment out the replication factor = 3 part. When I turn on the replication, the error message is – UnexpectedStatusCodeError: Collection may not have been created properly.! Unexpected status code: 422, with response body: {‘error’: [{‘message’: ‘init sharding state: not enough replicas: found 1 want 3’}]}.

We did successfully setup 3 node weaviate server in docker compose. Why there is a sharding involved? and is there anything we missed in configuration?

client.connect()

client.collections.delete(“GptLists”)

GptsVectors = client.collections.create(
name=“GptLists”,
# Not working yet
# replication_config=wvc.Configure.replication(
# factor=3
# ),
vectorizer_config=wvc.Configure.Vectorizer.text2vec_openai(
model=“text-embedding-3-small”,
),
generative_config=wvc.Configure.Generative.openai(),
inverted_index_config=wvc.Configure.inverted_index(
index_property_length=True
),
vector_index_config=wvc.Configure.VectorIndex.hnsw(),
properties=[
wvc.Property(name=“title”, data_type=wvc.DataType.TEXT),
wvc.Property(name=“description”, data_type=wvc.DataType.TEXT),
wvc.Property(name=“prompt_starters”, data_type=wvc.DataType.TEXT_ARRAY),
wvc.Property(name=“welcome_message”, data_type=wvc.DataType.TEXT),
wvc.Property(name=“categories”, data_type=wvc.DataType.TEXT, skip_vectorization=True),
wvc.Property(name=“language”, data_type=wvc.DataType.TEXT, skip_vectorization=True),
wvc.Property(name=“conversations”, data_type=wvc.DataType.INT, skip_vectorization=True),
],
)

Server Setup Information

  • Weaviate Version: 1.23.7
  • Deployment Method:
    Docker Compose
  • Multi Node? Number of Running Nodes:
    3 Node
  • Used Client Language and Version:
    Python Client V4

Any additional Information

2

Hi @Lawrence_Hope ! Welcome to our community!

Can you make sure you see all 3 nodes in your endpoint:
http://localhost:8080/v1/nodes

Looks like it may not be findinf the other 2 classes, given the error message.

3

This is what I have in response, looks like only see one node.
{“nodes”:[{“batchStats”:{“queueLength”:0,“ratePerSecond”:0},“gitHash”:“d5c2694”,“name”:“weaviate1”,“shards”:null,“stats”:{“objectCount”:0,“shardCount”:1},“status”:“HEALTHY”,“version”:“1.23.7”}]}
200

In the docker-compose file and the log output from weaviate, we did see THREE nodes running up in the SAME IP in SAME machine. But at different port. Here is the abbreviated docker compose:

weaviate-1:
command:
- --host
- 0.0.0.0
- --port
- ‘8080’
- --scheme
- http
image: semitechnologies/weaviate:1.23.7
init: true
ports:
- 8080:8080
- 50051:50051
CLUSTER_HOSTNAME: ‘weaviate1’
CLUSTER_GOSSIP_BIND_PORT: ‘7100’
CLUSTER_DATA_BIND_PORT: ‘7101’

weaviate-2:
command:
- --host
- 0.0.0.0
- --port
- ‘8080’
- --scheme
- http
image: semitechnologies/weaviate:1.23.7
init: true
ports:
- 8081:8080
- 50052:50051
CLUSTER_HOSTNAME: ‘weaviate2’
CLUSTER_GOSSIP_BIND_PORT: ‘7102’
CLUSTER_DATA_BIND_PORT: ‘7103’
CLUSTER_JOIN: ‘weaviate-node-1:7100’
weaviate-3:
command:
- --host
- 0.0.0.0
- --port
- ‘8080’
- --scheme
- http
image: semitechnologies/weaviate:1.23.7
init: true
ports:
- 8082:8080
- 50053:50051
CLUSTER_HOSTNAME: ‘weaviate3’
CLUSTER_GOSSIP_BIND_PORT: ‘7104’
CLUSTER_DATA_BIND_PORT: ‘7105’
CLUSTER_JOIN: ‘weaviate-node-1:7100’

By using 8081 and 8082 too. We got this output:
{“nodes”:[{“batchStats”:{“queueLength”:0,“ratePerSecond”:0},“gitHash”:“d5c2694”,“name”:“weaviate1”,“shards”:null,“stats”:{“objectCount”:0,“shardCount”:1},“status”:“HEALTHY”,“version”:“1.23.7”}]}
200
{“nodes”:[{“batchStats”:{“queueLength”:0,“ratePerSecond”:0},“gitHash”:“d5c2694”,“name”:“weaviate2”,“shards”:null,“stats”:{“objectCount”:0,“shardCount”:0},“status”:“HEALTHY”,“version”:“1.23.7”}]}
200
{“nodes”:[{“batchStats”:{“queueLength”:0,“ratePerSecond”:0},“gitHash”:“d5c2694”,“name”:“weaviate3”,“shards”:null,“stats”:{“objectCount”:0,“shardCount”:0},“status”:“HEALTHY”,“version”:“1.23.7”}]}
200

I think we are closed to find out true reasons. :wink:

4

That seems to be the issue.

Your weaviate-1 server is not aware of the other nodes.

Probably you can see some logs on that on the other clusters.

Can you change CLUSTER_JOINon the other containers should point to be like:

CLUSTER_JOIN: ‘weaviate-1:7100’

this should be the same name for your weaviate-1 service.

Let me know if this helps.

5

It works!!! :star_struck: Thank you so much.

And the output now is good and found 3 replicates: weaviate-1-1 | time=“2024-02-08T06:47:23Z” level=debug msg=“shard=ATQ75mAl0dOa is ready” action=startup
weaviate-1-1 | time=“2024-02-08T06:47:23Z” level=info msg=“Created shard gptlists_ATQ75mAl0dOa in 1.803349ms”
weaviate-1-1 | time=“2024-02-08T06:47:23Z” level=debug msg=“shard=bVpE1J0RJv8L is ready” action=startup
weaviate-1-1 | time=“2024-02-08T06:47:23Z” level=debug msg=“shard=ATQ75mAl0dOa is ready” action=startup
weaviate-1-1 | time=“2024-02-08T06:47:23Z” level=debug msg=“shard=KTUzZd8b6Y23 is ready” action=startup
weaviate-1-1 | time=“2024-02-08T06:47:23Z” level=debug msg=“prefilled level in vector cache” action=hnsw_vector_cache_prefill_level count=0 hnsw_level=0 index_id=main took=“83.773µs”
weaviate-1-1 | time=“2024-02-08T06:47:23Z” level=info msg=“prefilled vector cache” action=hnsw_vector_cache_prefill count=1000 index_id=main limit=1000000000000 took=“141.915µs”
weaviate-1-1 | time=“2024-02-08T06:47:23Z” level=debug msg=“received HTTP request” action=restapi_request method=GET url=/v1/schema/GptLists
weaviate-2-1 | time=“2024-02-08T06:47:34Z” level=debug msg=" memberlist: Initiating push/pull sync with: weaviate3 172.21.0.6:7104"
weaviate-3-1 | time=“2024-02-08T06:47:34Z” level=debug msg=" memberlist: Stream connection from=172.21.0.7:43294"
weaviate-3-1 | time=“2024-02-08T06:47:49Z” level=debug msg=" memberlist: Initiating push/pull sync with: weaviate2 172.21.0.7:7102"
weaviate-2-1 | time=“2024-02-08T06:47:49Z” level=debug msg=" memberlist: Stream connection from=172.21.0.6:35250"
weaviate-1-1 | time=“2024-02-08T06:47:51Z” level=debug msg=" memberlist: Initiating push/pull sync with: weaviate3 172.21.0.6:7104"
weaviate-3-1 | time=“2024-02-08T06:47:51Z” level=debug msg=" memberlist: Stream connection from=172.21.0.4:38106"
weaviate-3-1 | time=“2024-02-08T06:48:04Z” level=debug msg=" memberlist: Stream connection from=172.21.0.7:33010"
weaviate-2-1 | time=“2024-02-08T06:48:04Z” level=debug msg=" memberlist: Initiating push/pull sync with: weaviate3 172.21.0.6:7104"
weaviate-3-1 | time=“2024-02-08T06:48:19Z” level=debug msg=" memberlist: Initiating push/pull sync with: weaviate2 172.21.0.7:7102"
weaviate-2-1 | time=“2024-02-08T06:48:19Z” level=debug msg=" memberlist: Stream connection from=172.21.0.6:42494"
weaviate-3-1 | time=“2024-02-08T06:48:21Z” level=debug msg=" memberlist: Stream connection from=172.21.0.4:33954"
weaviate-1-1 | time=“2024-02-08T06:48:21Z” level=debug msg=" memberlist: Initiating push/pull sync with: weaviate3 172.21.0.6:7104"
weaviate-2-1 | time=“2024-02-08T06:48:34Z” level=debug msg=" memberlist: Initiating push/pull sync with: weaviate1 172.21.0.4:7100"
weaviate-1-1 | time=“2024-02-08T06:48:34Z” level=debug msg=" memberlist: Stream connection from=172.21.0.7:42088"

1 Like