# Not able to ingest the batches of data

**URL:** <https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998>\
**Category:** Support\
**Tags:** integration, python\
**Created:** [July 10, 2024, 5:43am UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998 "2024-07-10T05:43:48Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vipul\_Maheshwari](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/vipul_maheshwari/32/1912_2.png) [@Vipul\_Maheshwari](https://forum.weaviate.io/u/Vipul_Maheshwari)\
**Post date:** [July 10, 2024, 5:43am UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/1 "2024-07-10T05:43:48Z")

</div>

Hey Hi Guys, i am facing so many issues in adding the batch of data during the ingestion part in the weaviate. Tried all the things but there is not a single start to end script which can help me to ingest all the data in the batches…

I tried skimp with this link : [Batch import | Weaviate - Vector Database](https://weaviate.io/developers/weaviate/manage-data/import)

but nothing seems to be working.

Can anyone help me with it? I really need it…

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 10, 2024, 1:31pm UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/2 "2024-07-10T13:31:41Z")

</div>

hi @Vipul_Maheshwari !!

Welcome to our community.

Please, when opening a thread, fill in the requested info, like server version, deployment, etc.

Do you see any error logs? Can you share any code we can reproduce?

Thanks!

---

<div class="post-metadata">

**Author:** ![Vipul\_Maheshwari](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/vipul_maheshwari/32/1912_2.png) [@Vipul\_Maheshwari](https://forum.weaviate.io/u/Vipul_Maheshwari)\
**Post date:** [July 11, 2024, 12:52pm UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/3 "2024-07-11T12:52:27Z")

</div>

Hey @DudaNogueira thanks for reverting back.

From the next time, I will make sure to fill the requested info and other details.

So I have completed this script for batch ingestion, can you just skim through it fast and let me know if there is any kind of error in it:

```auto
import numpy as np
import logging
import time
import weaviate
from tqdm import tqdm
import weaviate.classes.config as wc

# Constants
COLLECTION_NAME = "weaviate_test_collection_part6"
NUM_BATCHES = 10
VECTORS_PER_BATCH = 100
VECTOR_SIZE = 1536

# Setup logging
logging.basicConfig(level=logging.INFO)

# Connect to Weaviate
client = weaviate.connect_to_embedded()

# Create Weaviate collection
weaviate_collection = client.collections.create(
    name=COLLECTION_NAME,
    properties=[
        wc.Property(name="item", data_type=wc.DataType.TEXT),
    ],
    vectorizer_config=None
)

# Define the batch generation function
def make_batches(num_batches, vectors_per_batch, vector_size):
    for i in range(num_batches):
        try:
            vectors = np.random.rand(vectors_per_batch, vector_size).astype(np.float32)
            vectors_list = vectors.tolist()
            items = [str(i * vectors_per_batch + j + 1) for j in range(vectors_per_batch)]
            batch = list(zip(items, vectors_list))
            logging.info(f"Successfully generated batch {i+1}/{num_batches}")
            yield batch
        except Exception as e:
            logging.error(f"Error in batch {i+1}: {str(e)}")
            raise

# Main processing loop
try:
    total_time = 0.0
    batch_times = []
    for _batch_index, _batch in enumerate(tqdm(make_batches(num_batches=NUM_BATCHES, vectors_per_batch=VECTORS_PER_BATCH, vector_size=VECTOR_SIZE), desc="Processing batches", total=NUM_BATCHES)):
        ct = 0
        with weaviate_collection.batch.fixed_size(VECTORS_PER_BATCH) as batch:
            
            batch_start_time = time.time()
            for item, vector in _batch:

                batch.add_object(
                    properties={"item": item},
                    vector=vector
                )

                ct += 1
                
                # If the number of vectors reached VECTORS_PER_BATCH threshold, it means the batch is injected with the desired number of vectors. (Ingestion of one batch is completed)
                if ct % VECTORS_PER_BATCH == 0:
                    duration = time.time() - batch_start_time
                    batch_times.append(duration)
                    total_time += duration
                    print(f"Processed {ct} vectors in batch {_batch_index + 1} of {NUM_BATCHES} in {duration:.2f}s")
    
    print(f"Total processing time: {total_time:.2f}s")
    print(f"Average time per batch: {np.mean(batch_times):.2f}s")

except Exception as e:
    logging.error(f"An error occurred during processing: {str(e)}")
    raise

finally:
    pass

```

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 15, 2024, 1:22pm UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/4 "2024-07-15T13:22:10Z")

</div>

Hi!

Can you try catching those errors?

Check here:

> **[Python | Weaviate - Vector Database](https://weaviate.io/developers/weaviate/client-libraries/python#error-handling)**
>
> Overview

---

<div class="post-metadata">

**Author:** ![Vipul\_Maheshwari](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/vipul_maheshwari/32/1912_2.png) [@Vipul\_Maheshwari](https://forum.weaviate.io/u/Vipul_Maheshwari)\
**Post date:** [July 21, 2024, 7:08am UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/5 "2024-07-21T07:08:37Z")

</div>

Hey Hi! I think I figured it out, can you just confirm if this sounds good to you, Thanks in advance:

```auto
import numpy as np
import logging
import time
import weaviate
from tqdm import tqdm
import weaviate.classes.config as wc

# Constants
COLLECTION_NAME = "weaviate_test_collection_part6"
NUM_BATCHES = 10
VECTORS_PER_BATCH = 100
VECTOR_SIZE = 1536

# Setup logging
logging.basicConfig(level=logging.INFO)
# Connect to Weaviate
client = weaviate.connect_to_embedded()

# Create Weaviate collection
weaviate_collection = client.collections.create(
    name=COLLECTION_NAME,
    properties=[
        wc.Property(name="item", data_type=wc.DataType.TEXT),
    ],
    vectorizer_config=None
)

# Define the batch generation function
def make_batches(num_batches, vectors_per_batch, vector_size):
    for i in range(num_batches):
        try:
            vectors = np.random.rand(vectors_per_batch, vector_size).astype(np.float32)
            vectors_list = vectors.tolist()
            items = [str(i * vectors_per_batch + j + 1) for j in range(vectors_per_batch)]
            batch = list(zip(items, vectors_list))
            logging.info(f"Successfully generated batch {i+1}/{num_batches}")
            yield batch
        except Exception as e:
            logging.error(f"Error in batch {i+1}: {str(e)}")
            raise

# Main processing loop
try:
    total_time = 0.0
    batch_times = []
    for _batch_index, _batch in enumerate(tqdm(make_batches(num_batches=NUM_BATCHES, vectors_per_batch=VECTORS_PER_BATCH, vector_size=VECTOR_SIZE), desc="Processing batches", total=NUM_BATCHES)):
        ct = 0
        with weaviate_collection.batch.fixed_size(VECTORS_PER_BATCH) as batch:
            
            batch_start_time = time.time()
            for item, vector in _batch:

                batch.add_object(
                    properties={"item": item},
                    vector=vector
                )

                ct += 1
                
                # If the number of vectors reached VECTORS_PER_BATCH threshold, it means the batch is injected with the desired number of vectors. (Ingestion of one batch is completed)
                if ct % VECTORS_PER_BATCH == 0:
                    duration = time.time() - batch_start_time
                    batch_times.append(duration)
                    total_time += duration
                    print(f"Processed {ct} vectors in batch {_batch_index + 1} of {NUM_BATCHES} in {duration:.2f}s")
    
    print(f"Total processing time: {total_time:.2f}s")
    print(f"Average time per batch: {np.mean(batch_times):.2f}s")

except Exception as e:
    logging.error(f"An error occurred during processing: {str(e)}")
    raise

finally:
    pass

```

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 23, 2024, 12:45pm UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/6 "2024-07-23T12:45:09Z")

</div>

Hi!

This seems fine.

Now the idea is that you can experiment with different batch sizes.

Here you can have more info on that:  
[https://weaviate-python-client.readthedocs.io/en/stable/weaviate.batch.html#module-weaviate.batch.crud\_batch](https://weaviate-python-client.readthedocs.io/en/stable/weaviate.batch.html#module-weaviate.batch.crud_batch)

Thanks!

---

<div class="post-metadata">

**Author:** ![Vipul\_Maheshwari](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/vipul_maheshwari/32/1912_2.png) [@Vipul\_Maheshwari](https://forum.weaviate.io/u/Vipul_Maheshwari)\
**Post date:** [July 23, 2024, 1:07pm UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/7 "2024-07-23T13:07:27Z")

</div>

Yes! But i wanted to take the VECTORS\_PER\_ BATCH variables as my go to for deciding the number of batches I want to ingest at a time…

Thanks for putting this through, sorry for the inconvenience! I am just glad you reviewed the snippet and I am good to go…

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 23, 2024, 1:10pm UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/8 "2024-07-23T13:10:53Z")

</div>

That’s ok.

You can probably get interesting results with dynamic, as it will adjust the batch size according to what the server reports back, taking into account the current server load.

---

<div class="post-metadata">

**Author:** ![Vipul\_Maheshwari](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/vipul_maheshwari/32/1912_2.png) [@Vipul\_Maheshwari](https://forum.weaviate.io/u/Vipul_Maheshwari)\
**Post date:** [July 23, 2024, 1:29pm UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/9 "2024-07-23T13:29:36Z")

</div>

Well to be honest, I am running a benchmark for various DBs to understand the time it takes for the ingestion as well as the bottleneck for the server load…

So it would be unfair to change the batch size dynamically

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 23, 2024, 1:31pm UTC](https://forum.weaviate.io/t/not-able-to-ingest-the-batches-of-data/2998/10 "2024-07-23T13:31:18Z")

</div>

Fair enough 🙂

You can also try enabling the [ASYNC\_INDEXING](https://weaviate.io/developers/weaviate/config-refs/schema/vector-index#asynchronous-indexing), so you don’t need to wait for the indexation step.