Broadcast_abort_transaction: broadcast tx abort failed

Support
1

Context

This happens while updating a collection. My weaviate instance is running on GKE and this is the 1st time I encountered the issue since last 15 days. I have two collections(Benchmark and Benchmark_test).

import weaviate
import weaviate.classes as wvc

client = weaviate.connect_to_custom(
    http_host="*.*.*.*",
    http_port=80,
    http_secure=False,
    grpc_host="*.*.*.*",
    grpc_port=50051,
    grpc_secure=False,
    )
articles = client.collections.get("Benchmark")

articles.aggregate.over_all(total_count=True)
# AggregateReturn(properties={}, total_count=11172946)

# articles = client.collections.get("Benchmark_test")
# articles.aggregate.over_all(total_count=True)
# AggregateReturn(properties={}, total_count=141195)

Update the collection

articles.config.update(
    replication_config=wvc.config.Reconfigure.replication(
        factor=2
    )
)

OR

articles.config.update(
    vector_index_config=wvc.config.Reconfigure.VectorIndex.hnsw(ef=256)
)

ERROR

{
action: "broadcast_abort_transaction"
error: "host "*.*.*.*:7001": send http request: Delete "http://*.*.*.*:7001/schema/transactions/ba832d25-3213-2322-13ds-3cefa2ewewc15": context deadline exceeded"
id: "ba832d25-3213-2322-13ds-3cefa2ewewc15"
level: "error"
msg: "broadcast tx abort failed"
}

Query is working fine but Ingestion also seems to have some issue. I see this messages in the log for all the props.

benchmark-ann-modified  | {'error': [{'message': "update inverted indices: put inverted indices props: no bucket for prop '**content**' found"}]}

Also tried:
additional_config = weaviate.config.AdditionalConfig(timeout=(300, 600))

The issue started flashing after I fired both the update command (replication_config and vector_index_config) in parallel. Did it lock something internally?

Looking forward getting help in understanding where did it get wrong and how can I fix it.

2

Hi @Aman_Ranjan_verma !

What are the versions of the server and client you are using?

Can you reproduce this in latest version?

Thanks!

3

I am using server version 16.8.5 and client version 4.4.1. Let me try today to reproduce the issue on the latest version of the server.

4

Hi @Aman_Ranjan_verma

Can you check the Weaviate server version?

you can get that information at

http://weaviate-server:8080/v1/meta

our current version is 1.24

Thanks!

5

This ticket can be closed. My incomplete findings.

16.8.5 was the helm chart version, I was using.

I tried bumping the helm version. Didn’t help
I tried deleting the helm release and reinstalling. PVC was not deleted, and upon reinstalling pods, they were trying to connect to the existing PVC. 2 out of 4 replicas were successful, but the third was failing.

Then, in my last iteration,. I deleted the release and manually deleted the PVC. After a fresh installation,. I lost my data (intended); everything was working fine and I couldn’t reproduce the issue.

1 Like