Description
How to use trust_remote_code=True
in case of locally downloaded gated model from huggingface to be used with Weaviate.
Server Setup Information
- Weaviate Server Version: 1.25.4
- Deployment Method: docker
- Multi Node? Number of Running Nodes: Single Node
- Client Language and Version: Python
I’m using Dockerfile as below to create the container of locally downloaded huggingface model:
$ cat Nvidia-NV-Embed.Dockerfile
FROM semitechnologies/transformers-inference:custom
# Copy the locally downloaded model to the Docker image
COPY local_NV-Embed-v1 /app/models/model
# Set the environment variable to trust remote code
ENV TRUST_REMOTE_CODE=True
and using docker-compose file as:
$ cat docker-compose.yml
---
services:
weaviate:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.25.4
ports:
- 8080:8080
- 50051:50051
volumes:
- /path/to/weaviate_data:/var/lib/weaviate
restart: on-failure:0
environment:
TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers:8080'
QNA_INFERENCE_API: 'http://qna-transformers:8080'
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
ENABLE_MODULES: 'text2vec-transformers,qna-transformers'
CLUSTER_HOSTNAME: 'node'
t2v-transformers:
image: nvidia-nv-embed-inference
environment:
ENABLE_CUDA: '0'
TRUST_REMOTE_CODE: 'true'
qna-transformers:
image: roberta_large_squad2_inference
environment:
ENABLE_CUDA: '0'
...
I’m using TRUST_REMOTE_CODE
at both places, still I’m getting this error:
$ sudo docker compose logs -f t2v-transformers
weaviate_docker_2-t2v-transformers-1 | INFO: Started server process [7]
weaviate_docker_2-t2v-transformers-1 | INFO: Waiting for application startup.
weaviate_docker_2-t2v-transformers-1 | INFO: CUDA_PER_PROCESS_MEMORY_FRACTION set to 1.0
weaviate_docker_2-t2v-transformers-1 | INFO: Running on CPU
weaviate_docker_2-t2v-transformers-1 | The repository for ./models/model contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/./models/model.
weaviate_docker_2-t2v-transformers-1 | You can avoid this prompt in future by passing the argument `trust_remote_code=True`.
weaviate_docker_2-t2v-transformers-1 |
weaviate_docker_2-t2v-transformers-1 | ERROR: Traceback (most recent call last):
weaviate_docker_2-t2v-transformers-1 | File "/usr/local/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 599, in resolve_trust_remote_code
weaviate_docker_2-t2v-transformers-1 | answer = input(
weaviate_docker_2-t2v-transformers-1 | ^^^^^^
weaviate_docker_2-t2v-transformers-1 | EOFError: EOF when reading a line
weaviate_docker_2-t2v-transformers-1 |
weaviate_docker_2-t2v-transformers-1 | During handling of the above exception, another exception occurred:
weaviate_docker_2-t2v-transformers-1 |
weaviate_docker_2-t2v-transformers-1 | Traceback (most recent call last):
weaviate_docker_2-t2v-transformers-1 | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 734, in lifespan
weaviate_docker_2-t2v-transformers-1 | async with self.lifespan_context(app) as maybe_state:
weaviate_docker_2-t2v-transformers-1 | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 610, in __aenter__
weaviate_docker_2-t2v-transformers-1 | await self._router.startup()
weaviate_docker_2-t2v-transformers-1 | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 713, in startup
weaviate_docker_2-t2v-transformers-1 | handler()
weaviate_docker_2-t2v-transformers-1 | File "/app/app.py", line 74, in startup_event
weaviate_docker_2-t2v-transformers-1 | meta_config = Meta(model_dir, model_name, use_sentence_transformer_vectorizer)
weaviate_docker_2-t2v-transformers-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weaviate_docker_2-t2v-transformers-1 | File "/app/meta.py", line 11, in __init__
weaviate_docker_2-t2v-transformers-1 | self.config = AutoConfig.from_pretrained(model_path).to_dict()
weaviate_docker_2-t2v-transformers-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weaviate_docker_2-t2v-transformers-1 | File "/usr/local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1141, in from_pretrained
weaviate_docker_2-t2v-transformers-1 | trust_remote_code = resolve_trust_remote_code(
weaviate_docker_2-t2v-transformers-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
weaviate_docker_2-t2v-transformers-1 | File "/usr/local/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 612, in resolve_trust_remote_code
weaviate_docker_2-t2v-transformers-1 | raise ValueError(
weaviate_docker_2-t2v-transformers-1 | ValueError: The repository for ./models/model contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/./models/model.
weaviate_docker_2-t2v-transformers-1 | Please pass the argument `trust_remote_code=True` to allow custom code to be run.
weaviate_docker_2-t2v-transformers-1 |
weaviate_docker_2-t2v-transformers-1 | ERROR: Application startup failed. Exiting.
weaviate_docker_2-t2v-transformers-1 exited with code 3
Pls help me how and where should I pass this argument trust_remote_code=True
in case of locally downloaded gated model from huggingface to be used with Weaviate ?