I'm having a problem with Redis Sentinel.
I deployed
on my Kubernetes environment the Redis HA mode with Sentinel offered by bitnami.
Up until a few days ago everything was working fine and I could connect to both sentinel container and redis. Since yesterday I've been getting this error in the log files
09:04:32.67 INFO ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD timeout 220 redis-cli -h redis.gntn-bacheca-gad.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster
the redis image I'm using is docker.io/bitnami/redis:7.0.8-debian-11-r0 while the sentinel one is docker.io/bitnami/redis-sentinel:7.0.7-debian-11-r10
This is the generated StatefulSet I'm using:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-node
namespace: "gntn-bacheca-gad"
labels:
app.kubernetes.io/name: redis
helm.sh/chart: redis-17.8.0
app.kubernetes.io/instance: redis
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: node
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: redis
app.kubernetes.io/instance: redis
app.kubernetes.io/component: node
serviceName: redis-headless
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: redis
helm.sh/chart: redis-17.8.0
app.kubernetes.io/instance: redis
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: node
annotations:
checksum/configmap: 8e45506deb5deafd4cda3e5c947463e9d9b8a9938c48cc0d184adfabb7507449
checksum/health: fe5ba0c879bdeefc22d3dfe886e8ef4ae9f34c7edc9cf747a6de5d06e1412318
checksum/scripts: 1dfa44150f64f3c63dfaa9e466197b88f1f47f5281ecef92258d8cd89cbe6987
checksum/secret: c5f840c872f7859ba66fd5333fe32304ecb92fa071ca93866c8fb318969f4d23
spec:
securityContext:
fsGroup: 1001
serviceAccountName: redis
affinity:
podAffinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: redis
app.kubernetes.io/instance: redis
app.kubernetes.io/component: node
topologyKey: kubernetes.io/hostname
weight: 1
nodeAffinity:
terminationGracePeriodSeconds: 30
containers:
- name: redis
image: docker.io/bitnami/redis:7.0.8-debian-11-r0
imagePullPolicy: "IfNotPresent"
lifecycle:
preStop:
exec:
command:
- /bin/bash
- -c
- /opt/bitnami/scripts/start-scripts/prestop-redis.sh
securityContext:
runAsUser: 1001
command:
- /bin/bash
args:
- -c
- /opt/bitnami/scripts/start-scripts/start-node.sh
env:
- name: BITNAMI_DEBUG
value: "false"
- name: REDIS_MASTER_PORT_NUMBER
value: "6379"
- name: ALLOW_EMPTY_PASSWORD
value: "no"
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis
key: redis-password
- name: REDIS_MASTER_PASSWORD
valueFrom:
secretKeyRef:
name: redis
key: redis-password
- name: REDIS_TLS_ENABLED
value: "no"
- name: REDIS_PORT
value: "6379"
- name: REDIS_SENTINEL_TLS_ENABLED
value: "no"
- name: REDIS_SENTINEL_PORT
value: "26379"
- name: REDIS_DATA_DIR
value: /data
ports:
- name: redis
containerPort: 6379
startupProbe:
failureThreshold: 22
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
exec:
command:
- sh
- -c
- /health/ping_liveness_local.sh 5
livenessProbe:
initialDelaySeconds: 20
periodSeconds: 5
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
exec:
command:
- sh
- -c
- /health/ping_liveness_local.sh 5
readinessProbe:
initialDelaySeconds: 20
periodSeconds: 5
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 5
exec:
command:
- sh
- -c
- /health/ping_readiness_local.sh 1
resources:
limits: {}
requests: {}
volumeMounts:
- name: start-scripts
mountPath: /opt/bitnami/scripts/start-scripts
- name: health
mountPath: /health
- name: redis-data
mountPath: /data
- name: config
mountPath: /opt/bitnami/redis/mounted-etc
- name: redis-tmp-conf
mountPath: /opt/bitnami/redis/etc
- name: tmp
mountPath: /tmp
- name: sentinel
image: docker.io/bitnami/redis-sentinel:7.0.7-debian-11-r10
imagePullPolicy: "IfNotPresent"
lifecycle:
preStop:
exec:
command:
- /bin/bash
- -c
- /opt/bitnami/scripts/start-scripts/prestop-sentinel.sh
securityContext:
runAsUser: 1001
command:
- /bin/bash
args:
- -c
- /opt/bitnami/scripts/start-scripts/start-sentinel.sh
env:
- name: BITNAMI_DEBUG
value: "false"
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis
key: redis-password
- name: REDIS_SENTINEL_TLS_ENABLED
value: "no"
- name: REDIS_SENTINEL_PORT
value: "26379"
ports:
- name: redis-sentinel
containerPort: 26379
startupProbe:
failureThreshold: 22
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
exec:
command:
- sh
- -c
- /health/ping_sentinel.sh 5
livenessProbe:
initialDelaySeconds: 20
periodSeconds: 5
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
exec:
command:
- sh
- -c
- /health/ping_sentinel.sh 5
readinessProbe:
initialDelaySeconds: 20
periodSeconds: 5
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 5
exec:
command:
- sh
- -c
- /health/ping_sentinel.sh 1
resources:
limits: {}
requests: {}
volumeMounts:
- name: start-scripts
mountPath: /opt/bitnami/scripts/start-scripts
- name: health
mountPath: /health
- name: sentinel-data
mountPath: /opt/bitnami/redis-sentinel/etc
- name: redis-data
mountPath: /data
- name: config
mountPath: /opt/bitnami/redis-sentinel/mounted-etc
volumes:
- name: start-scripts
configMap:
name: redis-scripts
defaultMode: 0755
- name: health
configMap:
name: redis-health
defaultMode: 0755
- name: config
configMap:
name: redis-configuration
- name: sentinel-data
emptyDir: {}
- name: redis-tmp-conf
emptyDir: {}
- name: tmp
emptyDir: {}
volumeClaimTemplates:
- metadata:
name: redis-data
labels:
app.kubernetes.io/name: redis
app.kubernetes.io/instance: redis
app.kubernetes.io/component: node
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"
Since it worked before I tried to remove the pods from kubernetes and to install them again but the problem persists. What could this problem be caused by?
