GRPC Query failed AioRpcError of RPC terminated status UNAVAILABLE

Description

Seeing this error pop up in our logs

2024-09-26T18:36:47.646Z
Error finding nearest neighbors: Query call with protocol GRPC search failed with message <AioRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = “recvmsg:Connection reset by peer”
debug_error_string = “UNKNOWN:Error received from peer {created_time:“2024-09-26T18:36:47.646280137+00:00”, grpc_status:14, grpc_message:“recvmsg:Connection reset by peer”}”

Server Setup Information

  • Weaviate Server Version: 1.26.4
  • Deployment Method: Weaviate Cloud
  • Multi Node? Number of Running Nodes: 1??
  • Client Language and Version: Python
  • Multitenancy?: No.

Any additional Information

Code where the error is populated:

def find_nearest_neighbors(item, settings):
vector_similarity = settings[“vector_similarity”]
matching_vectors = settings[“matching_vectors”]

try:
    description = item["description"]
    item_id = item["id"] if "id" in item else item["item_id"]
    embedding = get_embedding(description)
    collection = vector_client.collections.get("items")

    if embedding is None:
        raise ValueError("No embedding found for item")

    results = collection.query.near_vector(
        near_vector=embedding,
        certainty=vector_similarity,
        filters=Filter.by_property("item_id").not_equal(item_id),
        return_metadata=wvc.query.MetadataQuery(certainty=True),
        limit=matching_vectors
    )

    items_dict_list = [item.properties for item in results.objects]
    return items_dict_list

except Exception as e:
    print(f"Error finding nearest neighbors: {e}")
    return []

hi @AdamHeard !!

Welcome to our community :hugs:

Do you have any reading on resource usage?

How many of memory and cpu do you have available? How many objects have stored?

this error message indicates that the connection was closed from the server.

So I believe there may be requiring more resources.

If you have not fine tuned Weaviate for resource usage, check this documentation on how to do that:

Thanks for the response @DudaNogueira.

Per resources on the Fargate Instance:
CPU: 2 vCPU
Memory: 4 GB
Objects Stored: 1.6M

How much more resources do you feel it would require?

Hi!

This will vary according to the dimension length of your vectors.

Here we have an example for this calculation

Assuming your vectors has 1536 dimensions, this is the calculation you need to do, for storing 1 million objects:

2 * 1e6 * (1536 * 4)

As a rule of thumb, we always double the required memory (this is the 2x)

so you are looking to something around 12G of ram. This is not counting the maxconnections.

Let me know if this helps.