I am setting up a ELK stack on Kubernetes. Already did it one for pre-prod environment and it is working well. Unfortunately, when trying to duplicate the setup for a prod environment I came across this error :
Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
I have xpack security enabled and did setup passwords with bin/elasticsearch-setup-passwords CLI tool. I then set the generated elastic password in my elastic YAML and restarted my elastic deployment. But I still have this error showing up. I tried resetting the elastic password, same issue. I also have all the needed PVC and Services.
Here is the Kurbenetes YML deployment for my elastic master node if needed :
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "4"
kompose.cmd: kompose convert
kompose.version: 1.26.1 (a9d05d509)
objectset.rio.cattle.io/applied: H4sIAAAAAAAA/7RVTW/jNhD9KwueWkCmJflbt2C3hwLbNtgstociEEbUOGbNL5Ajb43A/70Y2Yk/2g2cpntROCHfmzeP9MyjgKC/YEzaO1EJCCENN4XIxFq7VlTiAwbjtxYdiUxYJGiBQFSPApzzBKS9SxyuvQ0+oVSWUYfonfJug5GhT/ub51SFLKeyePcDLNp80k7yxY8iE775ExUlJBm1lwqIDErth5pZR63Ki0KVg7bM1WA8nTQDKMajQbksZ8txq8bT8VLsMqEi9sI+a4uJwAZRuc6YTBho0PRytZdPihLGjVYoKoEpL17QsIK0EpWYj2BcNNO5gmY5KabT6WQ+m7XTdpGr+WiGRTnGtszLBStxYE+IOUoB+lxfYY3tAM16ECIOQvTtgDARg1JAxRojBqMVJFEVmUhoUJGPvGGB1OrjFbXsmI0iED5s+SRtA+99wt4h5GSENhheM+93vt63Xs3uxBzlHYF2GJOo/ngU6Db934PhP328ufv88/v69ubu7vffPn0QmdiA6XjLQqgNNL5udI0GEmkldtkzsvGe2LIgLVoft7Xxan0Cp9jh6XllukQYpXaaNJjaAoe18y2mExjrzzDlJX9G/0bQh0dA69Ua4+CwewpodVJ+g3ErE2Jbr3yii0z/TMJyLjPsLT2e+SuAWkujFbreerOsH9Ahv55W9i/niG0gndu2BydUXdS0leigMdi+4NsFYEUUZEpGKoykl1rxkzyi+b9pyIr7j1SRXklWQ0crHzXps3vZEysYKriW9L/XtsbtCzXx7hUkG4z7krR3tfXtpU1P5n2biiK4FHyk/83wbzK+2fVz5ldbfw5/i//nTFdfwn0mtIWH4y9aHnqOVH54WCaEqFbnUTWXpczF5QRhAfuG99z/bn0kUS3KPOdkEZPvosK+kxptNaV9Z+dexl05n41m42JejvtuuvGms/iL79yB1vLyFogH3bBLcZhWEPFCqfJuqR+GvYFHifuQHbyKpB80J/VxzA3p/lAGQaRbb7Ri2TfmK2z7F9QLTqe9/klG4NGTCB196Q+9N6BtPyp48euZyJN7fs58JcNR6e5+tx+wQB27vNv9HQAA//+wjN6eTQkAAA
objectset.rio.cattle.io/id: 3dc011c2-d20c-465b-a143-2f27f4dc464f
creationTimestamp: "2022-09-06T11:59:47Z"
generation: 4
labels:
io.kompose.service: es01
objectset.rio.cattle.io/hash: 83a41b68cabf516665877d6d90c837e124ed2029
managedFields:
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kompose.cmd: {}
f:kompose.version: {}
f:objectset.rio.cattle.io/applied: {}
f:objectset.rio.cattle.io/id: {}
f:labels:
.: {}
f:io.kompose.service: {}
f:objectset.rio.cattle.io/hash: {}
f:spec:
f:progressDeadlineSeconds: {}
f:replicas: {}
f:revisionHistoryLimit: {}
f:selector: {}
f:strategy:
f:type: {}
f:template:
f:metadata:
f:annotations:
.: {}
f:cattle.io/timestamp: {}
f:kompose.cmd: {}
f:kompose.version: {}
f:labels:
.: {}
f:io.kompose.service: {}
f:spec:
f:affinity: {}
f:containers:
k:{"name":"es01"}:
.: {}
f:env:
.: {}
k:{"name":"ELASTIC_PASSWORD"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:secretKeyRef:
.: {}
f:key: {}
f:name: {}
f:optional: {}
k:{"name":"ES_JAVA_OPTS"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"cluster.initial_master_nodes"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"cluster.name"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"discovery.seed_hosts"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"node.name"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.license.self_generated.type"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.enabled"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.http.ssl.certificate"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.http.ssl.certificate_authorities"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.http.ssl.enabled"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.http.ssl.key"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.http.ssl.verification_mode"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.transport.ssl.certificate"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.transport.ssl.certificate_authorities"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.transport.ssl.enabled"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.transport.ssl.key"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"xpack.security.transport.ssl.verification_mode"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:ports:
.: {}
k:{"containerPort":9200,"protocol":"TCP"}:
.: {}
f:containerPort: {}
f:name: {}
f:protocol: {}
f:resources:
.: {}
f:limits:
.: {}
f:memory: {}
f:requests:
.: {}
f:memory: {}
f:securityContext:
.: {}
f:allowPrivilegeEscalation: {}
f:capabilities: {}
f:privileged: {}
f:readOnlyRootFilesystem: {}
f:runAsUser: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/usr/share/elasticsearch/config/certs"}:
.: {}
f:mountPath: {}
f:name: {}
k:{"mountPath":"/usr/share/elasticsearch/data"}:
.: {}
f:mountPath: {}
f:name: {}
f:dnsPolicy: {}
f:nodeName: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext:
.: {}
f:runAsUser: {}
f:terminationGracePeriodSeconds: {}
f:volumes:
.: {}
k:{"name":"certs"}:
.: {}
f:name: {}
f:persistentVolumeClaim:
.: {}
f:claimName: {}
k:{"name":"esdata01"}:
.: {}
f:name: {}
f:persistentVolumeClaim:
.: {}
f:claimName: {}
manager: rancher
operation: Update
time: "2022-09-06T12:50:29Z"
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:deployment.kubernetes.io/revision: {}
f:status:
f:availableReplicas: {}
f:conditions:
.: {}
k:{"type":"Available"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Progressing"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
f:observedGeneration: {}
f:readyReplicas: {}
f:replicas: {}
f:updatedReplicas: {}
manager: kube-controller-manager
operation: Update
time: "2022-09-06T13:58:46Z"
name: es01
namespace: waked-elk-prod
resourceVersion: "219089408"
uid: 244c96dc-e6c6-4761-a0db-67bad33b94fa
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
io.kompose.service: es01
strategy:
type: Recreate
template:
metadata:
annotations:
cattle.io/timestamp: "2022-09-06T12:40:20Z"
kompose.cmd: kompose convert
kompose.version: 1.26.1 (a9d05d509)
creationTimestamp: null
labels:
io.kompose.service: es01
spec:
affinity: {}
containers:
- env:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
key: ELASTIC_PASSWORD
name: elastic-creds-prod
optional: false
- name: cluster.initial_master_nodes
value: es01,es02,es03
- name: cluster.name
value: docker-cluster
- name: discovery.seed_hosts
value: es02,es03
- name: node.name
value: es01
- name: xpack.license.self_generated.type
value: basic
- name: xpack.security.enabled
value: "true"
- name: xpack.security.http.ssl.certificate
value: certs/es01/es01.crt
- name: xpack.security.http.ssl.certificate_authorities
value: certs/ca/ca.crt
- name: xpack.security.http.ssl.enabled
value: "true"
- name: xpack.security.http.ssl.key
value: certs/es01/es01.key
- name: xpack.security.http.ssl.verification_mode
value: certificate
- name: xpack.security.transport.ssl.certificate
value: certs/es01/es01.crt
- name: xpack.security.transport.ssl.certificate_authorities
value: certs/ca/ca.crt
- name: xpack.security.transport.ssl.enabled
value: "true"
- name: xpack.security.transport.ssl.key
value: certs/es01/es01.key
- name: xpack.security.transport.ssl.verification_mode
value: certificate
- name: ES_JAVA_OPTS
value: -Xms2g -Xmx2g
image: docker.elastic.co/elasticsearch/elasticsearch:8.2.0
imagePullPolicy: IfNotPresent
name: es01
ports:
- containerPort: 9200
name: 9200tcp
protocol: TCP
resources:
limits:
memory: 10000Mi
requests:
memory: 5000Mi
securityContext:
allowPrivilegeEscalation: true
capabilities: {}
privileged: false
readOnlyRootFilesystem: false
runAsUser: 1000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/share/elasticsearch/config/certs
name: certs
- mountPath: /usr/share/elasticsearch/data
name: esdata01
dnsPolicy: ClusterFirst
nodeName: k8-worker-cpu-3.cines.fr
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 30
volumes:
- name: certs
persistentVolumeClaim:
claimName: certs
- name: esdata01
persistentVolumeClaim:
claimName: esdata01
I tried to curl localhost from the es01 container ( curl -u elastic:generated_password localhost:9200 ) but got curl: (52) Empty reply from server.
I can access to Kibana through a browser and connect with the elastic/generated_password without any problem. I even managed to ingest some data in Elastic (that I could view through Kibana Dev Tools) but I feel like I should fix this error anyway.
Does anyone has a lead on how to debug this ?
