Migration cause Weaviate Cloud cluster restart all the time

Support
1

Description

Hi, my team is attempting to migrate Weaviate Cloud data from version 1.25 to a 1.28 cluster. The core code of the migration script is as follows. We are using the Weaviate client v4.

When executing the migration, the following errors occur. We have tried using dynamic batch or fixed-sized batch settings with size=100 and concurrent_request=1, but there are still a large number of errors. After multiple errors, the cluster node will directly disconnect and restart.
with collection_tgt.batch.dynamic() as batch:
for q in collection_src.iterator(include_vector=True):
batch.add_object(
properties=q.properties,
vector=q.vector[“default”],
uuid=q.uuid
)

image
image1920×519 237 KB

We have 1M objects to migrate. Are there any alternative solutions to perform a safe data migration?

Server Setup Information

  • Weaviate Server Version: 1.25 → 1.28
  • Number of Running Nodes: 1
  • Client Language and Version: python, weaviate-client v4
  • Multitenancy?: No
2

Hello @becky,

Welcome to the community! It’s great to have you here.

I’ve received your ticket in Support and will follow up with a detailed reply there since you are a cloud customer and our ticketing system differs from the community forum. You could add the following snippet to your code to help identify the reason behind any failures:

failed_objects = client.batch.failed_objects
if failed_objects:
print(f"Number of failed objects: {len(failed_objects)}“)
for i, failed_obj in enumerate(failed_objects, 1):
print(f"Failed object {i}: {failed_obj}”)
else:
print(f"All objects successfully inserted into ‘{collection_name}’.")

This code will check for any objects that failed to be inserted and print out the reasons for the failures, helping you pinpoint what might be going wrong.

Regards,
Mohamed Shahin
Weaviate Support