Questions about using a local model with the reranker module In Python 3.10

Support
1

I’m trying to use a locally hosted model for reranking, but I’ve encountered some issues while following the official documentation that are preventing me from proceeding.

First, regarding the “Configure the integration” section for locally hosted rerankers.

Although this section is for locally hosted setups, the provided docker-compose.yml:

services:
  weaviate:
    # Other Weaviate configuration
    environment:
      RERANKER_INFERENCE_API: http://reranker-transformers:8080  # Set the inference API endpoint
  reranker-transformers:  # Set the name of the inference container
    image: cr.weaviate.io/semitechnologies/reranker-transformers:cross-encoder-ms-marco-MiniLM-L-6-v2
    environment:
      ENABLE_CUDA: 0  # Set to 1 to enable

Although, this chapter is “Locally Hosted Transformers Reranker Models with Weaviate”, but the example seems to reference an online model.

I would like to know how I should configure this if I have downloaded a model from Hugging Face to my local machine. Specifically:

Should the RERANKER_INFERENCE_API be changed to a local path?

Or, should the image value under reranker-transformers be modified to point to my local model files?

Are there any other environment variables I need to set to make Weaviate use the local model instead of trying to download one?

Secondly, in the “References” section, three “Available models” are listed:

cross-encoder/ms-marco-MiniLM-L-6-v2
cross-encoder/ms-marco-MiniLM-L-2-v2
cross-encoder/ms-marco-TinyBERT-L-2-v2

Is this the complete list of supported models? Or is it possible to use other reranking models from Hugging Face or other sources?

Ultimately, I would like to know how to modify the configuration to use a local model for reranking. It would be very helpful if you could show me how to do this, or provide a working example/case study for local model reranking.