0

I'm trying to deploy elasticsearch on kubernetes, with memory_lock enabled, but nothing works all the time. First, I'll paste my kubernetes config

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  labels:
    app: elasticsearch
spec:
  replicas: 1
  template:
    metadata:
      name: elasticsearch
      labels:
        app: elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: pawelcyrklaf/elastic:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: http.host
              value: 0.0.0.0
            - name: transport.host
              value: 0.0.0.0
            - name: xpack.ml.enabled
              value: "false"
            - name: xpack.security.enabled
              value: "false"
            - name: xpack.monitoring.enabled
              value: "false"
            - name: xpack.graph.enabled
              value: "false"
            - name: xpack.watcher.enabled
              value: "false"
            - name: cluster.name
              value: elasticsearch-monitoring
            - name: bootstrap.memory_lock
              value: "true"
            - name: ES_JAVA_OPTS
              value: -Xms4G -Xmx4G
            - name: discovery.zen.ping.unicast.hosts
              value: elasticsearch
            - name: node.master
              value: "true"
            - name: node.data
              value: "false"
            - name: node.ingest
              value: "false"
            - name: discovery.zen.minimum_master_nodes
              value: "1"
          ports:
            - containerPort: 9200
              name: http
            - containerPort: 9300
              name: http2
          resources:
            requests:
              memory: 2Gi
              cpu: 200m
            limits:
              memory: 4Gi
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - IPC_LOCK
                - SYS_RESOURCE
            privileged: true
            procMount: Default
            readOnlyRootFilesystem: false
            runAsNonRoot: false
      restartPolicy: Always
  selector:
    matchLabels:
      app: elasticsearch
---
apiVersion: v1
kind: Service
metadata:
  name: elastic-service
  labels:
    name: elastic-service
spec:
  selector:
    app: elasticsearch
  ports:
    - name: http
      port: 9200
      protocol: TCP
      targetPort: 9200
    - name: transport
      port: 9300
      protocol: TCP
      targetPort: 9300
  type: NodePort

pawelcyrklaf/elastic:latest - it's my custom image with below configuration

FROM docker.elastic.co/elasticsearch/elasticsearch:6.3.0

COPY elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml
COPY elasticsearch /etc/sysconfig/elasticsearch
COPY elasticsearch.service /usr/lib/systemd/system/elasticsearch.service
COPY elasticsearch.service /etc/systemd/system/elasticsearch.service.d/override.conf
RUN echo vm.max_map_count=262144 >> /etc/sysctl.conf
RUN echo fs.file-max=131072 >> /etc/sysctl.conf
COPY limits.conf /etc/security/limits.conf

EXPOSE 9200

elasticsearch.yml

bootstrap.memory_lock: true
discovery.type: single-node
xpack.security.enabled: false
http.host: 0.0.0.0
network.host: 0.0.0.0

elasticsearch

ES_JAVA_OPTS="-Xms4g -Xmx4g"  
MAX_LOCKED_MEMORY=unlimited

elasticsearch.service

[Service]
LimitMEMLOCK=unlimited

limits.conf

elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

Of course, I built the image, started deployment. All the files that are copied when building the image are in the pod, but memory_lock doesn't work anyway. In pod logs I have enter image description here

and

[7NlQMVt] memory locking requested for elasticsearch process but memory is not locked

I tried a newer version of elastisearch but that didn't work either. The problem is that every now and then elasticsearch eats up all the RAM and I have to add more RAM, although practically nothing happens on the application that uses elasticsearch.

I've read a lot of tutorials on google how to fix it, a lot of topics on stackoverflow, but nothing works for me. I tried to run this deployment on minikube, rancher and AWS EKS, but there is the same problem everywhere and I have no idea how to fix it :(

PawelC
  • 149
  • 1
  • 11
  • Not sure if related, but sysctls you're trying to set will do nothing because they are inside container and they seems to be at node-level, not namespaced. See https://docs.openshift.com/container-platform/4.8/nodes/containers/nodes-containers-sysctls.html Also: why would you copy systemd unit file when no systemd is involved at all when running application inside container? Seems like no reason to do so. – morsik Feb 03 '23 at 18:43

0 Answers0