Handling gRPC timeouts and vector index synchronization lags in high-concurrency mobile environments

General
1

Hi everyone,

I’ve been testing a hybrid search implementation using Weaviate for a mobile-first vector application, and I’ve run into a consistent performance bottleneck when the client device is under heavy load.

I’ve noticed that when my mobile environment is under high resource pressure—specifically when I’m running several background executors or testing the mobile app environment for its multi-threading capabilities—my gRPC calls to the Weaviate cluster frequently return DeadlineExceeded errors. Even more puzzling is that the local UI occasionally fails to render the results of a nearVector query, showing empty states or broken metadata placeholders, even though the backend logs show a successful retrieval. While I was discussing these environmental bottlenecks with some other developers, we noticed that the mobile OS seems to throttle the gRPC polling thread when the CPU load hits a specific threshold.

Has anyone else noticed that Weaviate clients become “sluggish” or start dropping UI assets on mobile if there are other high-performance tasks running in the background? I’m also seeing occasional vector drift where the results on mobile don’t match the desktop query perfectly, which makes me think the local index might be falling out of sync during these high-concurrency spikes.

I’m trying to figure out if I should be increasing the RAFT_TIMEOUTS_MULTIPLIER or if there’s a way to make the client-side hydration more resilient in “noisy” mobile environments. If you’ve dealt with stabilizing the mobile vector search experience for users who might be multitasking heavily—or have tips on ensuring UI consistency during heavy background processing—I’d really value your advice!

Best regards!

2

Good morning @Wehrli,

Welcome to our community! It’s lovely to have you here :innocent:

It’s really interesting to see a mobile-first vector application. Best of luck with the project, we’re glad you chose Weaviate as your Vector database.

However, I’d like to highlight a few points. You might already be aware of these, but based on my current knowledge (and I’m always open to learning if someone has found a perfect workaround), here is what I think:

DeadlineExceeded gRPC errors, these happen when the client-side timeout is shorter than the actual round-trip time to your Weaviate server. The standard workaround is to increase the timeout in your Weaviate client, though there is obviously a limit to how long you can ask a user to wait before the UX suffers.

If I assume your use case correctly, your mobile app does client requests to a Weaviate cluster. Since the HNSW index and all vector data live entirely on the server, as there isn’t a local index on the mobile device that would fall out of sync.

RAFT_TIMEOUTS_MULTIPLIER, this setting isn’t relevant to the error. It controls timeouts on the node-to-node network path. It’s used to manage Raft consensus timeouts (like Heartbeat, Election, and Leader Lease) to help cluster nodes tolerate internal network delays.

The issue is most likely just standard network latency. I’d suggest increasing the timeout, ensuring your Weaviate server has sufficient resources, and making sure your UI layer handles gRPC errors gracefully for the user.

Best regards,
Mohamed Shahin
Weaviate Support