Is it possible to turn off the use of any AVX instruction set(s) for Weaviate?

Support
1

I would like to test using Weaviate without any of the optimized SIMD setups discussed in: Distance metrics | Weaviate - Vector Database

I am getting deep… possibly kernel-level errors while building up vector graphs on my Linux computer and I’m trying to isolate the issue… I have had similar issues while using other vector DBs so I am confident that the crashes are related to these… Currently my machine goes through a full reboot cycle after building the graph up to around 1M vectors…

Is it possible to turn off the use of any AVX instruction set(s) for Weaviate ?
Is there any parameter I can pass to Weaviate or do I need to get into BIOS/chipset software and disable from there?

2

Hi @billy ! I will have to ask internally.

Can you share those errors?

Thanks!

1 Like
3

Hi @billy

One avenue would be trying to change GOARCH to a arch that doesn’t have this?

1 Like
4

Hi @billy,

This looks challenging (but fun) to debug particular if you are getting kernel level issues with multiple vector databases.

We use Go’s X86 sys/cpu feature detection for whether to use AVX instructions.

This is the relevant detection code for whether to use AVX for cosine / dot product (l2 is in l2_amd64.go). Setting GOARCH might disable this feature but as we use Go assembly you would need to check if this disabled. You could build Weaviate manually with this commented out to hard disable AVX instructions.

1 Like
5

HI thanks for the response !

The errors are really tricky to understand… but my unit hangs completely and reboots itself after a few minutes… sometimes there is a ‘Detected TX Unit Hang’ output on the monitor and other times it completely freezes…
I have a pretty beastly setup - its a Xeon 3rd Gen (Single Socket) running Linux 5.14.0-362.el9.x86_64
It supports all sorts of accelerated Vector Instruction sets - AVX/AVX2/AVX512…

I must say this has been really surprising Ive been using Weaviate for more than a year and I only very recently have come across this issue…

Edit: There is nothing on the Weaviate docker logs that looks like an error… So this comes from DEEP within the kernel I believe (and only during HNSW Graph Buildup which is vector distance compute intensive…)

6

Hi there thanks a lot for this !

I would like to try to building Weaviate manually without this specific code which checks for AVX2 !
Is there documentation on how to build Weaviate from source?
I have only used Weaviate with Docker compose before so this would be new but its worth a try I believe !

Edit: Is there a way to build Weaviate without these accelerations and create a new Docker image ? (would be really interesting)

7

Hi @billy - this guide covers it :slight_smile: Weaviate development setup | Weaviate - Vector Database.

It’s not been updated for some time, so please do let me know if you spot any errors. You can either do that here, or even create an issue (Issues · weaviate/weaviate-io · GitHub) on our repo.

1 Like