# Python client v4 batch create reference issue

**URL:** <https://forum.weaviate.io/t/python-client-v4-batch-create-reference-issue/1431>\
**Category:** Support\
**Tags:** bug, developer-experience, python\
**Created:** [February 8, 2024, 4:54pm UTC](https://forum.weaviate.io/t/python-client-v4-batch-create-reference-issue/1431 "2024-02-08T16:54:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Brian\_Money](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/brian_money/32/936_2.png) [@Brian\_Money](https://forum.weaviate.io/u/Brian_Money)\
**Post date:** [February 8, 2024, 4:54pm UTC](https://forum.weaviate.io/t/python-client-v4-batch-create-reference-issue/1431/1 "2024-02-08T16:54:09Z")

</div>

### Description

Python client weaviate-client==4.4.1

I have a schema of Documents and Chunks. I have a 2 way link Document-\>hasChunks, Chunk-\>ofDocument. The document I’m testing has 260 Chunks.

If I use client.batch.dynamic, all of the Document-\>hasChunks references are valid, but only about 40 of the Chunk-\>ofDocument are added, the rest are null. No errors are being reported.

If I use client.batch.fixed\_size(100), I get more Chunks with ofDocument (usually around 200)

If I use client.batch.fixed\_size(50, I get more Chunks (around 220)

I’m stumped.

Code:

```auto
with client.batch.dynamic() as batch:

    for index, element in enumerate(chunks):
        wv_chunk = Chunk.from_element(element, index)
        chunk_uuid= batch.add_object(
            collection="Chunk",
            properties=wv_chunk.get_data(),
        )
        print(f"Cross ref Doc id: {doc_uuid} hasChunks-> Chunk id: {chunk_uuid}")
        ref=batch.add_reference(
            from_collection="Document",
            from_uuid=doc_uuid,
            from_property="hasChunks",
            to=chunk_uuid
        )
        print(f"Cross ref Chunk id: {chunk_uuid} ofDocument-> Doc id: {doc_uuid}")
        ref = batch.add_reference(
            from_collection="Chunk",
            from_uuid=chunk_uuid,
            from_property="ofDocument",
            to=doc_uuid,
        )
        print('-' * 80)
  
failed_objs = client.batch.failed_objects
failed_refs = client.batch.failed_references
print(f"Failed batch objects: {failed_objs}")
print(f"Failed batch refs: {failed_refs}")
client.close()

```

Output:

```auto
Cross ref Doc id: 69438102-4464-4edb-a2d4-32887b5281e4 hasChunks-> Chunk id: 6e2c4288-5538-42c5-b525-0838329125f0
Cross ref Chunk id: 6e2c4288-5538-42c5-b525-0838329125f0 ofDocument-> Doc id: 69438102-4464-4edb-a2d4-32887b5281e4
--------------------------------------------------------------------------------
Cross ref Doc id: 69438102-4464-4edb-a2d4-32887b5281e4 hasChunks-> Chunk id: c3a53462-f70d-4290-b0b9-90869790cd8b
Cross ref Chunk id: c3a53462-f70d-4290-b0b9-90869790cd8b ofDocument-> Doc id: 69438102-4464-4edb-a2d4-32887b5281e4
--------------------------------------------------------------------------------
Failed batch objects: []
Failed batch refs: []

```

Now count the chunks with ofDocument

```auto
{
  Aggregate {
    Chunk(where: {
      operator: Equal,
      path: ["ofDocument","Document","id"],
      valueString: "69438102-4464-4edb-a2d4-32887b5281e4"
    }) {
			content {
        count
      }
    }
  }
}

{
  "data": {
    "Aggregate": {
      "Chunk": [
        {
          "content": {
            "count": 45
          }
        }
      ]
    }
  }
}

```

Here’s a run changing the code to client.batch.fixed\_size(50):

```auto
Cross ref Doc id: 69438102-4464-4edb-a2d4-32887b5281e4 hasChunks-> Chunk id: 0aa245c0-7cba-4623-91de-3936503def2d
Cross ref Chunk id: 0aa245c0-7cba-4623-91de-3936503def2d ofDocument-> Doc id: 69438102-4464-4edb-a2d4-32887b5281e4
--------------------------------------------------------------------------------
Cross ref Doc id: 69438102-4464-4edb-a2d4-32887b5281e4 hasChunks-> Chunk id: 269d625e-ef04-46a4-99f3-bcf2e63d54b1
Cross ref Chunk id: 269d625e-ef04-46a4-99f3-bcf2e63d54b1 ofDocument-> Doc id: 69438102-4464-4edb-a2d4-32887b5281e4
--------------------------------------------------------------------------------
Failed batch objects: []
Failed batch refs: []

```

Now count:

```auto
{
  Aggregate {
    Chunk(where: {
      operator: Equal,
      path: ["ofDocument","Document","id"],
      valueString: "69438102-4464-4edb-a2d4-32887b5281e4"
    }) {
			content {
        count
      }
    }
  }
}

{
  "data": {
    "Aggregate": {
      "Chunk": [
        {
          "content": {
            "count": 240
          }
        }
      ]
    }
  }
}

```

That was a good run.

Any idea what is going on?

### Server Setup Information

- Weaviate Server Version: 1.23.7
- Deployment Method: Weaviate cluster
- Multi Node? Number of Running Nodes:

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [February 8, 2024, 8:14pm UTC](https://forum.weaviate.io/t/python-client-v4-batch-create-reference-issue/1431/2 "2024-02-08T20:14:10Z")

</div>

Hi @Brian_Money ! Welcome to our community 🤗

if you import that same content using python v3 you get all objects? Trying to isolate here the py v4.

> No errors are being reported.

This in both on server and client, right?

Thanks!

---

<div class="post-metadata">

**Author:** ![Dirk](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dirk/32/46_2.png) [@Dirk](https://forum.weaviate.io/u/Dirk)\
**Post date:** [February 9, 2024, 3:23am UTC](https://forum.weaviate.io/t/python-client-v4-batch-create-reference-issue/1431/3 "2024-02-09T03:23:46Z")

</div>

This is something I sometimes noticed while developing the v4 client and I am 99% sure that it is a weaviate bug (reference batch add code is almost 100% copied from v3) . However I could not reliably reproduce it and am unsure about the cause.

We limited the ref batch size internally (regardless of user input) and that seemed to have fixed it in our testing, but might not have been enough ☹

Could you test with fixed\_size(50,1)? I have the suspicion that it is caused by the concurrency but am not really sure

---

<div class="post-metadata">

**Author:** ![Dirk](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dirk/32/46_2.png) [@Dirk](https://forum.weaviate.io/u/Dirk)\
**Post date:** [February 9, 2024, 7:15am UTC](https://forum.weaviate.io/t/python-client-v4-batch-create-reference-issue/1431/4 "2024-02-09T07:15:47Z")

</div>

Ok, I dug a bit more and I think I found the issue:

we are sending references too early, before weavaite has processed the from-object. In this case, the reference is silently discarded. Because weaviate does not return an error, we did not notice that wrong behaviour.

Not 100% sure how to solve this, but I’ll keep you up to date.

Until then:  
Try to first add all objects and then all references

---

<div class="post-metadata">

**Author:** ![Dirk](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dirk/32/46_2.png) [@Dirk](https://forum.weaviate.io/u/Dirk)\
**Post date:** [February 9, 2024, 1:03pm UTC](https://forum.weaviate.io/t/python-client-v4-batch-create-reference-issue/1431/5 "2024-02-09T13:03:21Z")

</div>

@Brian_Money Can you please try again with 4.4.3?