EKS: 1.21
Helm chart: loki-stack 2.6.5
Brand new installed loki not working …with this error:
```
$ kubectl get po | grep loki-0
loki-0 0/1 CrashLoopBackOff 7 11m
$ kubectl logs loki-0
level=info ts=2022-07-08T11:33:42.472156375Z caller=main.go:106 msg="Starting Loki" version="(version=2.5.0, branch=HEAD, revision=2d9d0ee23)"
level=info ts=2022-07-08T11:33:42.47257211Z caller=server.go:260 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-07-08T11:33:42.472664319Z caller=modules.go:597 msg="RulerStorage is not configured in single binary mode and will not be started."
level=info ts=2022-07-08T11:33:42.473112841Z caller=memberlist_client.go:394 msg="Using memberlist cluster node name" name=loki-0-3d8ddbac
level=warn ts=2022-07-08T11:33:42.473134806Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache"
level=info ts=2022-07-08T11:33:42.473520969Z caller=table_manager.go:169 msg="uploading tables"
level=info ts=2022-07-08T11:33:42.473556418Z caller=shipper_index_client.go:111 msg="starting boltdb shipper in 0 mode"
level=info ts=2022-07-08T11:33:42.47514934Z caller=modules.go:622 msg="RulerStorage is nil. Not starting the ruler."
level=info ts=2022-07-08T11:33:42.475970638Z caller=worker.go:112 msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
level=info ts=2022-07-08T11:33:42.478740368Z caller=module_service.go:64 msg=initialising module=server
level=info ts=2022-07-08T11:33:42.478767797Z caller=module_service.go:64 msg=initialising module=store
level=info ts=2022-07-08T11:33:42.478771179Z caller=module_service.go:64 msg=initialising module=memberlist-kv
level=error ts=2022-07-08T11:33:42.478826422Z caller=loki.go:384 msg="module failed" module=memberlist-kv error="invalid service state: Stopping, expected: Running"
level=error ts=2022-07-08T11:33:42.47886141Z caller=loki.go:384 msg="module failed" module=usage-report error="failed to start usage-report, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-07-08T11:33:42.478873521Z caller=loki.go:384 msg="module failed" module=compactor error="failed to start compactor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-07-08T11:33:42.478883112Z caller=loki.go:384 msg="module failed" module=distributor error="failed to start distributor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-07-08T11:33:42.47889261Z caller=loki.go:384 msg="module failed" module=ring error="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-07-08T11:33:42.478902985Z caller=loki.go:384 msg="module failed" module=query-scheduler error="failed to start query-scheduler, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-07-08T11:33:42.47891483Z caller=loki.go:384 msg="module failed" module=ingester-querier error="failed to start ingester-querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-07-08T11:33:42.478926647Z caller=loki.go:384 msg="module failed" module=ingester error="failed to start ingester, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-07-08T11:33:42.4789361Z caller=loki.go:384 msg="module failed" module=query-frontend-tripperware error="failed to start query-frontend-tripperware, because it depends on module server, which has failed: context canceled"
level=error ts=2022-07-08T11:33:42.478944908Z caller=loki.go:384 msg="module failed" module=querier error="failed to start querier, because it depends on module usage-report, which has failed: context canceled"
level=error ts=2022-07-08T11:33:42.478954773Z caller=loki.go:384 msg="module failed" module=query-frontend error="failed to start query-frontend, because it depends on module usage-report, which has failed: context canceled"
level=info ts=2022-07-08T11:33:42.478977421Z caller=table_manager.go:87 msg="stopping table manager"
level=info ts=2022-07-08T11:33:42.478991971Z caller=table_manager.go:169 msg="uploading tables"
level=info ts=2022-07-08T11:33:42.479012645Z caller=module_service.go:96 msg="module stopped" module=store
level=info ts=2022-07-08T11:33:42.480913383Z caller=modules.go:877 msg="server stopped"
level=info ts=2022-07-08T11:33:42.480933389Z caller=module_service.go:96 msg="module stopped" module=server
level=info ts=2022-07-08T11:33:42.480945833Z caller=loki.go:373 msg="Loki stopped"
level=error ts=2022-07-08T11:33:42.480995644Z caller=log.go:100 msg="error running loki" err="failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:419\nmain.main\n\t/src/loki/cmd/loki/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"
```
Helm values:
```
promtail:
enabled: false
fluent-bit:
enabled: true
loki:
persistence:
enabled: true
size: 1500Gi
isDefault: false
```
It works correctly on two more k8s clusters (1.18, 1.22) with the same configuration, but doesn't work particularly on this cluster.
Any advise how to debug or what could be wrong there.