# \[Question\] after Migrate data, something wrong, data lose

**URL:** <https://forum.weaviate.io/t/question-after-migrate-data-something-wrong-data-lose/21136>\
**Category:** Support\
**Tags:** technical\
**Created:** [May 22, 2025, 2:53am UTC](https://forum.weaviate.io/t/question-after-migrate-data-something-wrong-data-lose/21136 "2025-05-22T02:53:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jack\_Dim](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/jack_dim/32/2597_2.png) [@Jack\_Dim](https://forum.weaviate.io/u/Jack_Dim)\
**Post date:** [May 22, 2025, 2:53am UTC](https://forum.weaviate.io/t/question-after-migrate-data-something-wrong-data-lose/21136/1 "2025-05-22T02:53:50Z")

</div>

```auto
with collection.batch.fixed_size(batch_size=200) as batch:
    for data_row in data_rows:
        batch.add_object(
            properties=data_row,
        )

```

Hi, I’m from [SellerSprite](https://www.sellersprite.com/), currently I will build an app for amazon image searching. Thanks for your best open-source vector database. But something is wrong after migrating data (from a standalone to another cluster).

Over 11M objects, the standalone one indicate that those objects has 500G storage. but copied, the cluster only 10% disk usage.

I compared two total items:

```auto
.aggregate.over_all(total_count=True).total_count

```

these are same items. If execute docker commands, `docker compose down; docker compose up -d`, the cluster will show 3M objects.

Ok, where the rest 8M objects? Memory? How to fix this?

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [May 22, 2025, 1:52pm UTC](https://forum.weaviate.io/t/question-after-migrate-data-something-wrong-data-lose/21136/2 "2025-05-22T13:52:24Z")

</div>

hi @Jack_Dim !!

Welcome to our community 🤗

Have you check our migration guide?

> **[Migrate data | Weaviate](https://weaviate.io/developers/weaviate/manage-data/migrate)**
>
> Learn how to migrate data within Weaviate for easy data handling.

You should keep the id of the objects from source to target cluster.

That way you can retry the ingestion, and if the object is already ingested, it will be kept, due to the [deterministic id](https://weaviate.io/developers/weaviate/manage-data/create#generate-deterministic-ids)

Let me know if this helps!

---

<div class="post-metadata">

**Author:** ![Jack\_Dim](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/jack_dim/32/2597_2.png) [@Jack\_Dim](https://forum.weaviate.io/u/Jack_Dim)\
**Post date:** [May 27, 2025, 2:27am UTC](https://forum.weaviate.io/t/question-after-migrate-data-something-wrong-data-lose/21136/3 "2025-05-27T02:27:35Z")

</div>

Ok, thank you.  
Most time, the data will be lost when restart docker. sometimes when finishing copy, waiting for a long time, restart docker, the data will not be lost.

So, it’s Weird.

---

<div class="post-metadata">

**Author:** ![Jack\_Dim](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/jack_dim/32/2597_2.png) [@Jack\_Dim](https://forum.weaviate.io/u/Jack_Dim)\
**Post date:** [May 27, 2025, 12:24pm UTC](https://forum.weaviate.io/t/question-after-migrate-data-something-wrong-data-lose/21136/4 "2025-05-27T12:24:45Z")

</div>

I found the problem.

```auto
    volumes:
      - /db-500g/weaviate-node-3:/var/lib/weaviate
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 500
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'

```

THE docker insert files is not sync to `/db-500g/weaviate-node-3`, so the files’ size is not same.

So, how to force it to sync ?

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [May 27, 2025, 3:03pm UTC](https://forum.weaviate.io/t/question-after-migrate-data-something-wrong-data-lose/21136/5 "2025-05-27T15:03:22Z")

</div>

hi !

Not sure I understood the question.

You mean there is a difference in size between your nodes mount point for Weaviate data?

If that’s the case, it can happen due to some factors:  
multi tenancy: You may have bigger tenants on different nodes.  
out of sync collections: Due to how eventual consistency works, your nodes may be out of sync. [You can enable async repairs](https://weaviate.io/developers/weaviate/concepts/replication-architecture/consistency#async-replication), and they will sync data between nodes:

> **[Consistency | Weaviate](https://weaviate.io/developers/weaviate/concepts/replication-architecture/consistency)**
>
> The replication factor in Weaviate determines how many copies of shards (also called replicas) will be stored across a Weaviate cluster.

Let me know if this is the case here.

Thanks!