BUG: Batcher doesn't upload the objects to the weaviate db

Support
1

Description

In short, when I run automatic object uploader, that attempts to upload a set of objects one after another, my weaviate instance batcher doesn’t upload couple of provided sets. It returns a valid message “resp” in resp, err := batcher.Do(ctx) containing all of the fields and vectors for the provided object but doesn’t include the object in the database. Furthermore, when I upload large amounts of objects to the database, at some point I can’t even upload any object sets to the respective collection, but they do successfully upload to any other collection containing less elements.

Here is my code snippet:

batcher := client.Batch().ObjectsBatcher()

	// uploads data in batches of 20 or less chapters to prevent weaviate timeout
	batchCount := int(math.Ceil(float64(len(ObjectSet)) / batchSize))

	ObjectSetLen := len(ObjectSet)

	for i := 0; i < batchCount; i += 1 {
        // calculates the last object index for the object array that needs to be batched
		lastBatchElementIndex := (i + 1) * batchSize
		if lastBatchElementIndex > ObjectSetLen {
			lastBatchElementIndex = ObjectSetLen
		}

        // sets up the batcher and batches the objects
		for j := (i * batchSize); j < lastBatchElementIndex; j += 1 {
        //To avoid empty content error, makes a content check
			test := strings.ReplaceAll(ObjectSet[j].Content, " ", "")
			if test == "" {
				continue
			}
			dataObj := requests.SetSectionObj{
				Category:     ObjectSet[j].Title,
				Content:      ObjectSet[j].Content,
			}

			batcherObject := &models.Object{
				Class:      class,
				Properties: dataObj,
			}

			batcher.WithObjects(batcherObject)

		}

		resp, err := batcher.Do(ctx)
		if err != nil {
			utils.SetError(ctx, fasthttp.StatusInternalServerError, err.Error())
			return
		}
		log.Print(resp)
	}

	ctx.StatusCode(fasthttp.StatusNoContent)

Server Setup Information

  • Weaviate Server Version: 1.25.19
  • Deployment Method: Docker
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Go 1.23
  • Multitenancy?: No

Any additional Information

Weaviate logs do not provide any information in regards to this and does not throw any errors

2

hi @Jekabsons !!

Sorry, we missed this message :frowning:

Were you able to overcome this?

Thanks!

3

Hi, yes, the problem in the end was that I fetched the objects using this method:

resp, err := client.GraphQL().Get().WithClassName("ABCD").Do(context.Background())

And as it turns out I wasn’t familiar with pagination concept before and didn’t now that the maximum amount of objects a GraphQL query return
is 100. But I fixed the issue by using .WithOffset(100) method, and looping through the whole dataset.

1 Like
4

Thanks for sharing!!