Issue with Duplicate UUID Handling in Weaviate Batch Import vs. Insert Method

General
1

I’m experiencing an issue with duplicate UUID handling in Weaviate when using batch import.

In my batch import code, I generate a deterministic UUID using generate_uuid5(guid) for each object. This ensures that the same GUID consistently produces the same UUID. However, despite this, the batch import doesn’t seem to detect duplicates.

Here’s a simplified version of my code:

with collection.batch.fixed_size(batch_size=100) as batch:
    for _, row in processed_data.iterrows():
        for guid in row["GUIDs"]:
            try:
                batch.add_object(
                    properties={
                        "GUID": guid,
                        "a": row["a"]
                    },
                    vector={
                        key + "_embeddings": embeddings_dict.get(
                            row[key], [0.0] * 1536)
                        for key in ["a"]
                    },
                    uuid=generate_uuid5(guid)
                )
                records_processed += 1
            except weaviate.exceptions.UnexpectedStatusCodeError as e:
                skipped_details.append(
                    {
                        "GUID": guid,
                        "message": "Duplicate GUID found"
                        if e.status_code == 422
                        else str(e),
                    }
                )

When I run this code, it processes records with duplicate GUIDs without raising any errors. However, when I use the insert method with the same UUID generation logic, it correctly identifies duplicates and raises a 422 Unprocessable Entity error.

Why does the batch import not detect duplicates while the insert method does? Am I missing something in the batch import process?

Any insights or suggestions would be greatly appreciated.

1 Like
2

Good morning @Rohini_vaidya,

Your observation is right, when using batch import in Weaviate, adding objects with the same UUID does not raise a duplicate error in the same way as the insert method does. This is expected behavior. When you use batch import, if an object with a given UUID already exists, the new object will replace the existing one. This means that the batch import will silently overwrite the previous object with the same UUID, rather than raising an error:

Best regards,
Mohamed Shahin
Support Engineer – Weaviate
(Ireland, GMT/UTC timezone)

1 Like
3

Thanks for the clarification. I hadn’t realized that batch import intentionally overwrites existing objects with the same UUID instead of returning a duplicate error. That’s an important distinction, especially when building ingestion pipelines where accidental overwrites could happen. For projects that require strict duplicate detection, would you recommend validating UUIDs before batch import, or is there a more efficient pattern commonly used in production? I’d be interested to hear how others handle this.

4

Hey @matthew,

Batch import in Weaviate is upsert-by-design (same UUID = overwrite, no error). If you need strict reject duplicates, common production patterns (Deterministic UUIDs)

  1. Use the client’s collection.data.exists(uuid) or query by ID list (Filter.by_id().contains_any([…])) to filter out already-present UUIDs before sending the batch.

  2. If you truly need a per-object error. POST /objects (create, not batch) returns 422 on duplicate. However you lose batch throughput so it only worth it for low-volume.

Kind regards,
Mohamed Shahin
Tech Lead - Weaviate Support
(Ireland :ireland:, UTC+00:00/+01:00)
mohamed@weaviate.io
Weaviate & LinkedIn

1 Like