I have 4 million vectors, but the text2vec docker module (multi-qa-MiniLM-L6-cos-v1) seems to fail often on my setup. Usually, Out of memory eventually. I want to continue the database but manually add the vector after creating it locally.
My question is how do I vectorize locally to match the txt2vec module (that already setup a massive database)? Ie: is there a clear way how to do this locally without running the docker module locally?
I have a schema and it is clearly specific on what is included or not in txt2vec index, I was hoping there is documentation showing how: using the schema class how to locally embed (similar to txt2vec module).
Considering you have you can get the vectors directly from the model, you can add them to the database along with it. That way you can vectorize your data directly, and provide those vectors only when adding the object.
That’s what I have been doing. The docker instance routinely fails (my guess is that it falls behind the batch uploads for some reason (timeouts). I also get RAM (NVIDIA) space errors.).
File "C:\Users\M\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1378, in getresponse
response.begin()
File "C:\Users\M\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\M\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
I know how to set the vector on upload, but I want to match my “generated” vector (using the same model) to the one generated by text2vec.
ie: what is the simple way to generate the same vector
Does your Docker instance run fine except for occasional timeouts on import? One thing you can try to tune there is the batch size - to make them smaller for example.
Thanks - that document was what I was searching for! Looked for a while.
I have tried quite a few things - it (the text2vec module) I am guessing is the biggest hurdle and thing slowing down or causing issues. Even buying a semi-decent server with gpu wasn’t helping. Perhaps if I paid much more.
Although my manual embedding has some rando, timeouts, it seems to in general recover after auto retry.
[ERROR] Batch ConnectionError Exception occurred! Retrying in 2s. [1/3]
And this is acceptable for me especially since my test on manual uploads is on a low tier server for testing efficiency.
Depending on the system setup, the vectorization step can definitely be the bottleneck. Depending on what you need, you could try different things - for example this vectorizer uses one that’s based on a cpu-optimized architecture, but it might not have the same model available as the one you’ve used for the others.
Yes - had cuda setup and on. Did try the cpu text2vec-contextionary (cpu) - but never tried gpt4all approach. I think it is a combination of embedding speed and vector database size extending too large causing too many timeouts. My current solution is to increase timeout limit.
