HI Everyone, I’m Emmanuel Katto from Dubai, United Arab Emirates (UAE) We’re currently using PyTorch and the Transformers library in a custom Vectorizer class with the model sentence-transformers/all-MiniLM-L6-v2
to vectorize SQL, Python, and YAML files before sending them to Weaviate Cloud. However, this setup relies on a resource-intensive PyTorch and Transformers stack.
Has anyone successfully set up a similar process directly within Weaviate Cloud, or found a more lightweight alternative for vectorization? Any suggestions for optimizing or simplifying this workflow would be greatly appreciated!
Regards
Emmanuel Katto