# EKS Multi-Replica Raft Migration Failure in Weaviate 1.25.0

**URL:** <https://forum.weaviate.io/t/eks-multi-replica-raft-migration-failure-in-weaviate-1-25-0/22033>\
**Category:** Support\
**Tags:** k8s\
**Created:** [October 13, 2025, 8:44pm UTC](https://forum.weaviate.io/t/eks-multi-replica-raft-migration-failure-in-weaviate-1-25-0/22033 "2025-10-13T20:44:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bharath.go](https://avatars.discourse-cdn.com/v4/letter/b/8edcca/32.png) [@bharath.go](https://forum.weaviate.io/u/bharath.go)\
**Post date:** [October 13, 2025, 8:44pm UTC](https://forum.weaviate.io/t/eks-multi-replica-raft-migration-failure-in-weaviate-1-25-0/22033/1 "2025-10-13T20:44:25Z")

</div>

### Description

Hello, followed this [doc](https://docs.weaviate.io/deploy/migration/weaviate-1-25) to upgrade from Weaviate 1.22.6 to 1.25.0 on a 3-replica cluster resulted in incomplete Raft schema migration, causing 7 out of 19 classes to become inaccessible and failing with “shard not found” errors. The issue only occurs on multi-replica clusters. Single-replica on staging upgraded successfully. No errors in pod logs during upgrade - pods started successfully, no crashes.

### Server Setup Information

- **Weaviate Server Version:** 1.22.6 (attempted upgrade to 1.25.0, rolled back)
- **Deployment Method:** Helm chart on AWS EKS Fargate
- **Multi Node? Number of Running Nodes:** Yes, 3 nodes (StatefulSet with 3 replicas)
- \*\*Client Language and Version:\*\*Python client v3
- **Multitenancy?:** Yes - 5 out of 19 classes use multi-tenancy with 23-39 tenants each

### Additional Information

Environment Details  
Platform: AWS EKS Fargate  
Storage: AWS EFS with persistent volumes  
ReplicationFactor: 1 (no data redundancy across nodes)

```auto
image:
  registry: docker.io
  tag: 1.22.6
  repo: semitechnologies/weaviate
  pullPolicy: IfNotPresent
  pullSecrets: []

command: ["/bin/weaviate"]
args:
  - '--host'
  - '0.0.0.0'
  - '--port'
  - '8080'
  - '--scheme'
  - 'http'
  - '--config-file'
  - '/weaviate-config/conf.yaml'
  - --read-timeout=120s 
  - --write-timeout=120s

initContainers:
  sysctlInitContainer:
    enabled: true
    sysctlVmMaxMapCount: 524288
    image:
      registry: docker.io
      repo: alpine
      tag: latest
      pullPolicy: IfNotPresent
  
  extraInitContainers: {}

# 3-replica cluster configuration
replicas: 3

# Resource configuration
resources:
  requests:
    cpu: '16000m'
    memory: '64Gi'
  limits:
    cpu: '16000m'
    memory: '80Gi'

securityContext: {}

serviceAccountName:

# Persistent storage using AWS EFS
storage:
  size: 100Gi
  storageClassName: "efs-sc"

# Service configuration (NodePort for internal ALB)
service:
  name: weaviate
  ports:
    - name: http
      protocol: TCP
      port: 80
  type: NodePort
  loadBalancerSourceRanges: []
  clusterIP:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/cross-zone-load-balancing-enabled: "true"
    alb.ingress.kubernetes.io/scheme: internet-facing
    # Subnet IDs redacted
    service.beta.kubernetes.io/aws-load-balancer-subnets: <REDACTED>
    service.beta.kubernetes.io/aws-load-balancer-type: "ip"

# Probes configuration
startupProbe:
  enabled: false
  initialDelaySeconds: 300
  periodSeconds: 60
  failureThreshold: 50
  successThreshold: 1
  timeoutSeconds: 3

livenessProbe:
  initialDelaySeconds: 900
  periodSeconds: 10
  failureThreshold: 30
  successThreshold: 1
  timeoutSeconds: 3

readinessProbe:
  initialDelaySeconds: 120
  periodSeconds: 10
  failureThreshold: 10
  successThreshold: 1
  timeoutSeconds: 15

terminationGracePeriodSeconds: 150

# Weaviate Authentication Configuration
authentication:
  apikey:
    enabled: true
    allowed_keys:
      - '<REDACTED_API_KEY_1>'
      - '<REDACTED_API_KEY_2>'
    users:
      - admin@example.com
      - readonly@example.com
  anonymous_access:
    enabled: false
  oidc:
    enabled: false

# Authorization Configuration
authorization:
  admin_list:
    enabled: true
    users:
      - admin@example.com
    readonly_users:
      - readonly@example.com

query_defaults:
  limit: 100

debug: false

# Environment variables
env:
  CLUSTER_GOSSIP_BIND_PORT: 7000
  CLUSTER_DATA_BIND_PORT: 7001
  
  # Aggressive GC settings for memory management
  GOGC: 50
  LIMIT_RESOURCES: true

  # Prometheus metrics enabled
  PROMETHEUS_MONITORING_ENABLED: true

  # GOMEMLIMIT set to 60GB (64424509440 bytes)
  # Note: This is critical for preventing OOM kills
  GOMEMLIMIT: "64424509440"

  # Query limits
  QUERY_MAXIMUM_RESULTS: 15000

  # Vector dimension tracking disabled for performance
  TRACK_VECTOR_DIMENSIONS: false
  REINDEX_VECTOR_DIMENSIONS_AT_STARTUP: false

envSecrets: {}

# Backup providers (all disabled)
backups:
  filesystem:
    enabled: false
  s3:
    enabled: false
  gcs:
    enabled: false
  azure:
    enabled: false

# Modules configuration - all disabled (no vectorization modules)
modules:
  text2vec-contextionary:
    enabled: false
  text2vec-transformers:
    enabled: false
  text2vec-openai:
    enabled: false
  text2vec-huggingface:
    enabled: false
  text2vec-cohere:
    enabled: false
  text2vec-palm:
    enabled: false
  ref2vec-centroid:
    enabled: false
  multi2vec-clip:
    enabled: false
  qna-transformers:
    enabled: false
  qna-openai:
    enabled: false
  generative-openai:
    enabled: false
  generative-cohere:
    enabled: false
  generative-palm:
    enabled: false
  img2vec-neural:
    enabled: false
  reranker-cohere:
    enabled: false
  reranker-transformers:
    enabled: false
  text-spellcheck:
    enabled: false
  ner-transformers:
    enabled: false
  sum-transformers:
    enabled: false
  
  # No default vectorizer - using external embedding services
  default_vectorizer_module: none

custom_config_map:
  enabled: false
  name: 'custom-config'

annotations:

nodeSelector:

tolerations:

# Pod anti-affinity to spread replicas across nodes
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        podAffinityTerm:
          topologyKey: "kubernetes.io/hostname"
          labelSelector:
            matchExpressions:
              - key: "app"
                operator: In
                values:
                  - weaviate

```

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [October 14, 2025, 1:32pm UTC](https://forum.weaviate.io/t/eks-multi-replica-raft-migration-failure-in-weaviate-1-25-0/22033/2 "2025-10-14T13:32:14Z")

</div>

Hi!

We do not recommend jumping versions.

So for this migration, you will need:  
1.22.6 → restrt → 1.22.latest → restart → 1.23.latest → restart → 1.24.latest

Note that this is a significant migration. A lot has changed since 1.22

Depending on your dataset size, it’s probably easier to spin a new cluster and migrate your data over, using this migration guide: [Migrate data | Weaviate Documentation](https://weaviate.io/developers/weaviate/manage-data/migrate)

Let me know if this helps!