ObjectsGetResponse "double" error handling

General
1

Hello, I am just a bit confused about Weaviate’s choices in error handling within object response. Specifically, when attempting to flush a batch.

I had the following code (in golang):

	response, err := batcher.Do(ctx) // batcher is *batch.ObjectsBatcher
	if err != nil {
		return nil, err
	}

And everything worked well up until I had made a mistake, by providing a bit different object which didn’t abide by my originally set up BatchObject properties, thus causing an error. In the case above I would have expected to see an error in err variable that would terminate the program. However, it didn’t really happen. Instead I had to spend a significant amount of time to discover that I need to iterate through the response objects to identify the error since it didn’t appear anywhere:

	responses, err := batcher.Do(ctx)
	if err != nil {
		return nil, err
	}

	for _, resp := range responses {
		for _, errItem := range resp.Result.Errors.Error {
			if errItem != nil {
				log.Println("Batch error message:", errItem.Message)
			}
		}
	}

Why such choice of “double” error handling is needed in weaviate?

Seems very inefficient from development side, that just makes the code less readable. Furthermore, I didn’t really see anywhere in documentation addressing this issue of how to properly handle such errors, but it may be my fault of not being careful enough.

I would expect that the error handling is included within the .Do(context) method, just so the development is easier in future.

2

Hi @Jekabsons, thanks for your question. Agreed – the behavior is not immediately obvious, and, perhaps, somewhat unexpected.

By their nature, batch operations may complete/fail partially (IIRC that’s the only place you’d find such error handling), in which case you might want to retry inserting failed objects.
In contrast, the err returned from Do(ctx) means the operation has failed entirely (due to a network issue, malformed request, etc).

That being said, I totally see your point – to properly handle possible batch insert errors one’d need to iterate over response objects every time, even if the entire batch completed successfully. What would make more sense is having a custom PartialInsertError type (or sth to that extent) with a slice of objects that failed, so you could handle the errors like so:

r, err := batch.Do(ctx)
if err != nil {
    if partial, ok := err.(PartialInsertError); ok {
        for _, obj := range err.Failed {
            // insert again
        }
    } 
}

// happy path -- all objects inserted successfully

What do you think?

I didn’t really see anywhere in documentation addressing this issue of how to properly handle such errors

Yes, I just double-checked our Batch Import docs and seems that this section is missing for all clients but the latest Python v4. We’ll add an example of correct error handling to make sure this doesn’t catch others off-guard.

2 Likes
3

Thanks for the explenation. But just to be completely clear about your suggested implementation: if I do write a custom PartialInsertError type then I also need to write wrapper function for batch.Do(ctx) right that returns such error? Since the error is not present by default there.