Weaviate performance unatable by vectordbbenchmark evaluate

Support
,
1

Description

I ran Weaviate locally using its binary distribution and employed the Vector Database Benchmark Tool (VectorDBBench) to evaluate performance. After loading data with the command:

vectordbbench weaviate --url http://xx.xx.xx.xx:8080 --case-type Performance768D10M --db-label weaviate_test --no-auth --ef 512 --num-concurrency 96

I executed multiple test iterations ​with the --skip-load parameter​ to bypass data reloading. Despite identical configurations, the observed Queries Per Second (QPS) exhibited significant volatility across runs (e.g., 1900, 1700, 2000, 1600). ​What factors could contribute to this performance inconsistency?​

Server Setup Information

  • Weaviate Server Version: 1.31.1
  • Deployment Method:
  • Multi Node? Number of Running Nodes:
  • Client Language and Version:
  • Multitenancy?:

Any additional Information

2

hi @hemist !!

Welcome to our community :hugs:

I am not familiar with this tool ( VectorDBBench ) that I suppose is this one:

It is using our old client, and not benefitting from recent GRPC improvements.

For getting to the bottom of this, it is interesting to monitor some metrics and inspect logs to check for something out of ordinary.

Also, using our latest client is a must, as it will bring a lot of improvements.

Let me know if this helps!

THanks!