Description
I am trying to set up a RAG system with dockerized Weaviate and Ollama using a node JS backend, but am having trouble with the Ollama modules. When Weaviate tries to make a call to Ollama, it gives this error: WeaviateQueryError: Query call with protocol gRPC failed with message: /weaviate.v1.Weaviate/Search UNKNOWN: explorer: get class: concurrentTargetVectorSearch): explorer: get class: vectorize search vector: vectorize params: vectorize params: vectorize keywords: remote client vectorize: send POST request: Post "http://localhost:11434/api/embeddings": dial tcp [::1]:11434: connect: connection refused. This happens in both the vectorizer and generator, and both while loading data or searching it.
I have set up both Weaviate and Ollama to start in the same docker-compose. In my collections configuration, I have tried several different api endpoints - http://ollama:11434, http://localhost:11434, and http://host.docker.internal:11434. I have confirmed that Ollama is accessible at localhost:11434 and that I can use the models through both http and the js ollama library.
I’m assuming I’ve just made some mistake in the configuration here, but I’ve spent the better part of 2 days on this problem and no solution I’ve found works. Does anyone have any insights here?
Collection configuration
async createCollection(collectionName, properties) {
const exists = this.client.collections.exists(collectionName)
if (!exists) {
await this.client.collections.create({
name: collectionName,
properties: properties,
vectorizers: vectorizer.text2VecOllama({
apiEndpoint: 'http://ollama:11434',
model: 'nomic-embed-text',
}),
generative: generative.ollama({
apiEndpoint: 'http://ollama:11434',
model: 'llama3.1:8b',
}),
})
}
}
docker-compose.yml (I’m pretty sure OLLAMA_URL, OLLAMA_MODEL, and OLLAMA_EMBED_MODEL are for Verba which I am not using, but I included them just in case)
services:
weaviate:
command:
- --host
- 0.0.0.0
- --port
- '8765'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.26.4
ports:
- 8765:8765
- 50051:50051
volumes:
- weaviate_data:/var/lib/weaviate
container_name: weaviate
restart: on-failure:0
environment:
OLLAMA_URL: http://ollama:11434
OLLAMA_MODEL: llama3.1:8b
OLLAMA_EMBED_MODEL: nomic-embed-text:latest
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'text2vec-ollama'
ENABLE_MODULES: 'text2vec-ollama,generative-ollama'
CLUSTER_HOSTNAME: 'node1'
ollama:
image: ollama/ollama
ports:
- 11434:11434
volumes:
- ollama_data:/root/.ollama
- ./entrypoint.sh:/entrypoint.sh
container_name: ollama
pull_policy: always
tty: true
restart: always
entrypoint: ["/bin/bash", "/entrypoint.sh"]
volumes:
weaviate_data:
ollama_data:
entrypoint.sh is just a script to pull my ollama models once the container starts.
Server Setup Information
- Weaviate Server Version: 1.26.4
- Deployment Method:
- Multi Node? Number of Running Nodes: no
- Client Language and Version: JS API v3
- Multitenancy?: no
Any additional Information
In case it matters, I originally had it set up with text2vec-transformers and that part worked fine, which is how I know what happens when there is data in the db. I can upload a version of the dockerfile I used for that if it would be relevant.
I did also briefly run Verba to see if it was able to use the ollama integration, and I confirmed at least that it could see the models I had installed. I didn’t test further than that though.
Side note, I have also had trouble with deleting collections. When I try, it seems to partially but not fully delete it, and puts it into a weird state where I can’t recreate the collection because it think one still exists, but I can’t fully delete them either unless I clear the volume. I haven’t thoroughly tested this yet though because I’ve been focused on the Ollama problem.
