Getting "no object found for doc id" when creating new objects suddenly

junbetterway · February 6, 2024, 2:36pm

We are getting below error when creating new object:

WeaviateError(statusCode=500, messages=[WeaviateErrorMessage(message=put object: import into index profile: put local object: shard=“gV04uD1HB1QD”: Validate vector index for [105 35 162 129 86 191 78 64 176 10 54 87 7 235 243 174]: no object found for doc id 2241: no object for doc id, it could have been deleted, throwable=null)])

Note that we have been using Weaviate last year and it is the first time we encountered this error and could not find something related to it

So any help will be great - below are the details of our setup:

Weaviate version: 1.21.9
We are not feeding the object with our own vector embeddings but instead using Weaviate supported vectorizer module: semitechnologies/transformers-inference:sentence-transformers-multi-qa-MiniLM-L6-cos-v1

Below is the /nodes API check:

{
    "nodes": [
        {
            "batchStats": {
                "queueLength": 0,
                "ratePerSecond": 0
            },
            "gitHash": "621586d",
            "name": "OBSCURED_FOR_SECURITY",
            "shards": [
                {
                    "class": "Profile",
                    "name": "gV04uD1HB1QD",
                    "objectCount": 2375
                }
            ],
            "stats": {
                "objectCount": 2375,
                "shardCount": 1
            },
            "status": "HEALTHY",
            "version": "1.21.9"
        }
    ]
}

Our DevOps team provided below repeating Weaviate logs during the start of incident:

{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (1485) without a tombstone, tombstone was added","node_id":1485,"time":"2024-02-06T15:11:42Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (1352) without a tombstone, tombstone was added","node_id":1352,"time":"2024-02-06T15:11:42Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (2245) without a tombstone, tombstone was added","node_id":2245,"time":"2024-02-06T15:11:42Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (1041) without a tombstone, tombstone was added","node_id":1041,"time":"2024-02-06T15:11:42Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (2806) without a tombstone, tombstone was added","node_id":2806,"time":"2024-02-06T15:11:42Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (3518) without a tombstone, tombstone was added","node_id":3518,"time":"2024-02-06T15:11:42Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (3276) without a tombstone, tombstone was added","node_id":3276,"time":"2024-02-06T15:11:42Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (2232) without a tombstone, tombstone was added","node_id":2232,"time":"2024-02-06T15:11:42Z"}
{"action":"attach_tombstone_to_deleted_node","level":"info","msg":"found a deleted node (3479) without a tombstone, tombstone was added","node_id":3479,"time":"2024-02-06T15:11:42Z"}

Thank you

Just an update - it got fixed after upgrading to v1.23.7 (latest) but we will really appreciate the reason why it got broke all of a sudden without any changes on our side

Also, after the upgrade the /nodes results is now:

{
    "nodes": [
        {
            "batchStats": {
                "queueLength": 0,
                "ratePerSecond": 0
            },
            "gitHash": "d5c2694",
            "name": "OBSCURED_FOR_SECURITY",
            "shards": null,
            "stats": {
                "objectCount": 2380,
                "shardCount": 1
            },
            "status": "HEALTHY",
            "version": "1.23.7"
        }
    ]
}

DudaNogueira · February 6, 2024, 5:58pm

Hi @junbetterway !

Glad the new version fixed it. I will ask internally for someone with more visibility on this kind of issues.

Thanks!

junbetterway · February 7, 2024, 2:03am

Thank you - yeah but the management is worried a bit in case this recur so more of root cause analysis on our end. So appreciate for any findings.

Lastly, just let me know if what needs to be checked (e.g., needed logs). Thanks

jeronimo_irazabal · February 10, 2024, 3:10pm

Hi @junbetterway,

I did a review of the source code at version 1.21.9 and I found that in that version during object insertion, the vector dimension was validated to be the same as already inserted ones. But such comparison was made over a cache, which could lead to this “no object found” error.

That validation was changed since then and the comparison is made over a dimension property without querying the cache any more.

Ref lines:

github.com

weaviate/weaviate/blob/648f099d66a5053f75746af8b788205c3a0f02e0/adapters/repos/db/vector/hnsw/insert.go#L35


      
          
          func (h *hnsw) ValidateBeforeInsert(vector []float32) error {
          	dims := int(atomic.LoadInt32(&h.dims))
          
          	// no vectors exist
          	if dims == 0 {
          		return nil
          	}
          
          	// check if vector length is the same as existing nodes
          	if dims != len(vector) {
          		return fmt.Errorf("new node has a vector with length %v. "+
          			"Existing nodes have vectors with length %v", len(vector), dims)
          	}
          
          	return nil
          }
          
          func (h *hnsw) AddBatch(ctx context.Context, ids []uint64, vectors [][]float32) error {
          	if err := ctx.Err(); err != nil {
          		return err

github.com

weaviate/weaviate/blob/621586dacd3fe74de9219e130f4557d24259a10c/adapters/repos/db/vector/hnsw/insert.go#L31


      
          	"github.com/pkg/errors"
          	"github.com/weaviate/weaviate/adapters/repos/db/helpers"
          	"github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/distancer"
          )
          
          func (h *hnsw) ValidateBeforeInsert(vector []float32) error {
          	if h.isEmpty() {
          		return nil
          	}
          	// check if vector length is the same as existing nodes
          	existingNodeVector, err := h.cache.get(context.Background(), h.entryPointID)
          	if err != nil {
          		return err
          	}
          
          	if len(existingNodeVector) != len(vector) {
          		return fmt.Errorf("new node has a vector with length %v. "+
          			"Existing nodes have vectors with length %v", len(vector), len(existingNodeVector))
          	}
          
          	return nil

junbetterway · February 16, 2024, 9:46am

Hello @jeronimo_irazabal - thank you for lookin into this. This makes sense then since as per our devops engineer they perform server patches then probably required some restarts.

2020ashish · March 13, 2024, 5:47am

Hello @jeronimo_irazabal
On version v1.23.7
I’m getting this error again.
currently working with Python 3.
{“action”:“attach_tombstone_to_deleted_node”,“level”:“info”,“msg”:“found a deleted node (0) without a tombstone, tombstone was added”,“node_id”:0,“time”:“2024-03-13T04:48:16Z”}

DudaNogueira · March 13, 2024, 9:35pm

hi @2020ashish !

This is a info message. So nothing to worry AFAIK.

pnutmath · May 15, 2024, 4:17am

Related to attach_tombstone_to_deleted_node info message. is there any best practice that we need to follow. it looks like warning to me while reading the message

DudaNogueira · May 16, 2024, 1:57pm

hi @pnutmath !

This is a info level log that will report it’s action only.

Topic		Replies	Views
Help Needed: Resolving WeaviateQueryError with Nil or Zero-Length Vector at docID 715 Support	18	1914	May 11, 2024
Error Encountered in Weaviate Vector Search Support bug , python	1	1001	March 25, 2024
Weaviate "update vector: unmarshal error response body: Not Found" on Object Creation with text2vec-huggingface Support bug , technical	5	692	May 15, 2025
Debugging an insertion failure Support	3	967	April 9, 2024
Update inverted indices: put inverted indices props:no bucket for prop '<property_name>' found Support	7	623	September 25, 2024

Getting "no object found for doc id" when creating new objects suddenly

Related topics