Weaviate running in AWS VPC giving Error

[sasankasahu]

Description#####

Weaviate is not running and giving error below:

{“action”:“config_load”,“config_file_path”:“/weaviate-config/conf.yaml”,“level”:“info”,“msg”:“Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.”,“time”:“2024-04-01T13:09:41Z”}
{“deprecation”:{“apiType”:“Configuration”,“id”:“config-files”,“locations”:[“–config-file=""”],“mitigation”:“Configure Weaviate using environment variables.”,“msg”:“use of deprecated command line argument --config-file”,“sinceTime”:“2020-09-08T09:46:00.000Z”,“sinceVersion”:“0.22.16”,“status”:“deprecated”},“level”:“warning”,“msg”:“use of deprecated command line argument --config-file”,“time”:“2024-04-01T13:09:41Z”}
{“action”:“startup”,“default_vectorizer_module”:“none”,“level”:“info”,“msg”:“the default vectorizer modules is set to "none", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer”,“time”:“2024-04-01T13:09:41Z”}
{“action”:“startup”,“auto_schema_enabled”:true,“level”:“info”,“msg”:“auto schema enabled setting is set to "true"”,“time”:“2024-04-01T13:09:41Z”}
{“action”:“memberlist_init”,“bind_port”:7000,“error”:“Failed to get final advertise address: No private IP address found, and explicit IP not provided”,“hostname”:“”,“level”:“error”,“msg”:“memberlist not created”,“time”:“2024-04-01T13:09:41Z”}
{“action”:“startup”,“error”:“create member list: Failed to get final advertise address: No private IP address found, and explicit IP not provided”,“level”:“error”,“msg”:“could not init cluster state”,“time”:“2024-04-01T13:09:41Z”}

Server Setup Information

• Weaviate Server Version: 16.8.1
• Deployment Method: Helm Chart
• Multi Node? Number of Running Nodes: Running as container

Any additional Information

{“action”:“config_load”,“config_file_path”:“/weaviate-config/conf.yaml”,“level”:“info”,“msg”:“Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.”,“time”:“2024-04-01T13:09:41Z”}
{“deprecation”:{“apiType”:“Configuration”,“id”:“config-files”,“locations”:[“–config-file=""”],“mitigation”:“Configure Weaviate using environment variables.”,“msg”:“use of deprecated command line argument --config-file”,“sinceTime”:“2020-09-08T09:46:00.000Z”,“sinceVersion”:“0.22.16”,“status”:“deprecated”},“level”:“warning”,“msg”:“use of deprecated command line argument --config-file”,“time”:“2024-04-01T13:09:41Z”}
{“action”:“startup”,“default_vectorizer_module”:“none”,“level”:“info”,“msg”:“the default vectorizer modules is set to "none", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer”,“time”:“2024-04-01T13:09:41Z”}
{“action”:“startup”,“auto_schema_enabled”:true,“level”:“info”,“msg”:“auto schema enabled setting is set to "true"”,“time”:“2024-04-01T13:09:41Z”}
{“action”:“memberlist_init”,“bind_port”:7000,“error”:“Failed to get final advertise address: No private IP address found, and explicit IP not provided”,“hostname”:“”,“level”:“error”,“msg”:“memberlist not created”,“time”:“2024-04-01T13:09:41Z”}
{“action”:“startup”,“error”:“create member list: Failed to get final advertise address: No private IP address found, and explicit IP not provided”,“level”:“error”,“msg”:“could not init cluster state”,“time”:“2024-04-01T13:09:41Z”}

[jphwang]

Hi there,

Welcome and thanks for posting. It would be really helpful if you could post some additional information.

• Are you running Weaviate on a local K8s cluster, or on a cloud provider?
• Which instruction are you following, if any?
• Did this initially run, and then fail? Or did this happen the first time that you tried to run?
• Which configuration options did you change from the initial helm chart values.yaml?

[sasankasahu]

Hello there,
Please find more information on this.

• Are you running Weaviate on a local K8s cluster, or on a cloud provider?

I am running Weaviate in Openshift cluster which is running in Azure cloud.

• Which instruction are you following, if any?
  I am deploying the helm chart.

helm upgrade --install weaviate -f weavite.yaml weaviate/weaviate --version 16.8.1 --debug

• Did this initially run, and then fail? Or did this happen the first time that you tried to run?

It run successfully for 8 days then suddenly giving errors.

• Which configuration options did you change from the initial helm chart values.yaml?

No change from values file. It suddenly starts giving issue. Please find attached below for the values file.

COMPUTED VALUES:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- weaviate
topologyKey: kubernetes.io/hostname
weight: 1
annotations: null
args:

• –host
• 0.0.0.0
• –port
• “8080”
• –scheme
• http
• –config-file
• /weaviate-config/conf.yaml
• –read-timeout=60s
• –write-timeout=60s
  authentication:
  apikey:
  enabled: true
  allowed_keys:
  • f157a2ed-a1ce-4d65-a71e-cc7de22273a2
    users:
  • admin
    anonymous_access:
    enabled: false
    oidc:
    enabled: false
    authorization:
    admin_list:
    enabled: false
    users:
  • admin
    backups:
    azure:
    enabled: false
    envSecrets: {}
    envconfig:
    BACKUP_AZURE_CONTAINER: weaviate-backups
    secrets: {}
    filesystem:
    enabled: true
    envconfig:
    BACKUP_FILESYSTEM_PATH: /tmp/backups
    gcs:
    enabled: false
    envSecrets: {}
    envconfig:
    BACKUP_GCS_BUCKET: weaviate-backups
    secrets: {}
    s3:
    enabled: false
    envSecrets: {}
    envconfig:
    BACKUP_S3_BUCKET: weaviate-backups
    secrets: {}
    serviceAccountName: null
    command:
• /bin/weaviate
  custom_config_map:
  enabled: false
  name: custom-config
  debug: false
  env:
  CLUSTER_DATA_BIND_PORT: 7001
  CLUSTER_GOSSIP_BIND_PORT: 7000
  GOGC: 100
  PROMETHEUS_MONITORING_ENABLED: false
  QUERY_MAXIMUM_RESULTS: 100000
  REINDEX_VECTOR_DIMENSIONS_AT_STARTUP: false
  TRACK_VECTOR_DIMENSIONS: false
  envSecrets: null
  image:
  pullPolicy: IfNotPresent
  pullSecrets:
  registry: docker.io
  repo: semitechnologies/weaviate
  tag: 1.21.0
  initContainers:
  extraInitContainers: {}
  sysctlInitContainer:
  enabled: true
  image:
  pullPolicy: IfNotPresent
  registry: docker.io
  repo: alpine
  tag: latest
  sysctlVmMaxMapCount: 524288
  livenessProbe:
  failureThreshold: 30
  initialDelaySeconds: 900
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 3
  modules:
  default_vectorizer_module: none
  generative-cohere:
  apiKey: “”
  enabled: false
  generative-openai:
  apiKey: “”
  azureApiKey: “”
  enabled: false
  generative-palm:
  apiKey: “”
  enabled: false
  img2vec-neural:
  affinity: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: img2vec-neural
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/img2vec-pytorch
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: resnet50
  tolerations: null
  multi2vec-bind:
  annotations: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: clip-inference
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/multi2vec-bind
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: imagebind
  tolerations: null
  multi2vec-clip:
  annotations: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: clip-inference
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/multi2vec-clip
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: sentence-transformers-clip-ViT-B-32-multilingual-v1
  tolerations: null
  ner-transformers:
  affinity: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: ner-transformers
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/ner-transformers
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: dbmdz-bert-large-cased-finetuned-conll03-english-0.0.2
  tolerations: null
  qna-openai:
  apiKey: “”
  azureApiKey: “”
  enabled: false
  qna-transformers:
  affinity: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: qna-transformers
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/qna-transformers
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: bert-large-uncased-whole-word-masking-finetuned-squad-34d66b1
  tolerations: null
  ref2vec-centroid:
  enabled: false
  reranker-cohere:
  apiKey: “”
  enabled: false
  reranker-transformers:
  affinity: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: reranker-transformers
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/reranker-transformers
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: cross-encoder-ms-marco-MiniLM-L-6-v2
  tolerations: null
  sum-transformers:
  affinity: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: sum-transformers
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/sum-transformers
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: facebook-bart-large-cnn-1.0.0
  tolerations: null
  text-spellcheck:
  affinity: null
  enabled: false
  fullnameOverride: text-spellcheck
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/text-spellcheck-model
  resources:
  limits:
  cpu: 500m
  memory: 500Mi
  requests:
  cpu: 400m
  memory: 400Mi
  securityContext: {}
  serviceAccountName: null
  tag: pyspellchecker-en
  tolerations: null
  text2vec-cohere:
  apiKey: “”
  enabled: false
  text2vec-contextionary:
  affinity: null
  enabled: false
  envconfig:
  enable_compound_splitting: false
  extensions_storage_mode: weaviate
  neighbor_occurrence_ignore_percentile: 5
  occurrence_weight_linear_factor: 0.75
  fullnameOverride: contextionary
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  nodeSelector: null
  registry: docker.io
  replicas: 1
  repo: semitechnologies/contextionary
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 500m
  memory: 500Mi
  securityContext: {}
  serviceAccountName: null
  tag: en0.16.0-v1.0.2
  tolerations: null
  text2vec-gpt4all:
  affinity: null
  enabled: false
  fullnameOverride: gpt4all-inference
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/gpt4all-inference
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: all-MiniLM-L6-v2
  tolerations: null
  text2vec-huggingface:
  apiKey: “”
  enabled: false
  text2vec-openai:
  apiKey: “”
  azureApiKey: “”
  enabled: false
  text2vec-palm:
  apiKey: “”
  enabled: false
  text2vec-transformers:
  affinity: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: transformers-inference
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  passageQueryServices:
  passage:
  affinity: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: transformers-inference-passage
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/transformers-inference
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  tag: facebook-dpr-ctx_encoder-single-nq-base
  tolerations: null
  query:
  affinity: null
  enabled: false
  envconfig:
  enable_cuda: false
  ld_library_path: /usr/local/nvidia/lib64
  nvidia_driver_capabilities: compute,utility
  nvidia_visible_devices: all
  fullnameOverride: transformers-inference-query
  imagePullSecrets:
  inferenceUrl: {}
  livenessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  timeoutSeconds: 3
  nodeSelector: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/transformers-inference
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  tag: facebook-dpr-question_encoder-single-nq-base
  tolerations: null
  readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 3
  registry: docker.io
  replicas: 1
  repo: semitechnologies/transformers-inference
  resources:
  limits:
  cpu: 1000m
  memory: 5000Mi
  requests:
  cpu: 1000m
  memory: 3000Mi
  securityContext: {}
  serviceAccountName: null
  tag: distilbert-base-uncased
  tolerations: null
  nodeSelector: null
  query_defaults:
  limit: 100
  readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 3
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 3
  replicas: 1
  resources: {}
  securityContext: {}
  service:
  annotations: {}
  clusterIP: null
  loadBalancerSourceRanges:
  name: weaviate
  ports:
  • name: http
    port: 80
    protocol: TCP
    type: LoadBalancer
    serviceAccountName: null
    startupProbe:
    enabled: false
    failureThreshold: 50
    initialDelaySeconds: 300
    periodSeconds: 60
    successThreshold: 1
    timeoutSeconds: 3
    storage:
    size: 32Gi
    storageClassName: “”
    terminationGracePeriodSeconds: 600
    tolerations: null

The service controls how weaviate gRPC endpoint is exposed to the outside world.

If you don’t want a public load balancer, you can also choose ‘ClusterIP’ to make

weaviate gRPC port be only accessible within your cluster.

grpcService:

Set this to true in order to deploy Weaviate gRPC service

enabled: false
name: weaviate-grpc
ports:
- name: grpc
protocol: TCP
port: 50051

Target port is going to be the same for every port

type: NodePort
loadBalancerSourceRanges:

optionally set cluster IP if you want to set a static IP

clusterIP:
annotations: {}

[sasankasahu]

@jphwang any update on this as our client is waiting for a solution?

[jphwang]

Hi there @sasankasahu - it looks as though this is a problem with the k8s cluster and the vpc.

Pods are getting IP addresses assigned that are out of valid private address range.

• Reference: (Kubernetes | Weaviate - Vector Database)
• Also see (Memberlist failed to get final advertise address: No private IP address found, and explicit IP not provided - #2 by hebkon)

Since you mentioned it was working before, perhaps something caused the pods to restart, and possibly get assigned new IP addresses?