Elasticsearch with Kubernetes: Memory lock with bootstrap.memory_lock does not work

Question

I'm currently struggling with the k8s setup for our used elasticsearch with version 2.4.6:

Disable the memory swapping with bootstrap.memory_lock does not work. Memory reservation is failing with the well known error:

[2020-05-22 21:12:22,762][WARN ][bootstrap                ] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory
[2020-05-22 21:12:22,764][WARN ][bootstrap                ] This can result in part of the JVM being swapped out.
[2020-05-22 21:12:22,765][WARN ][bootstrap                ] Increase RLIMIT_MEMLOCK, soft limit: 83968000, hard limit: 83968000
[2020-05-22 21:12:22,765][WARN ][bootstrap                ] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited

I basically tried to follow the guidelines from this source: Memory Settings but it's still not properly running.

Any idea how to solve this issue?

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: elasticsearch
  name: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - image: elasticsearch:2.4.6
        imagePullPolicy: ""
        name: elasticsearch
        env:
         - name: ES_JAVA_OPTS
           value: "-Xmx512m -Xms512m"
         - name: ES_HEAP_SIZE
           value: "1g"
         - name: bootstrap.memory_lock
           value: "true"
        ports:
        - containerPort: 9200
        - containerPort: 9300
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          subPath: elasticsearch.yml
          name: elasticsearch-config
      initContainers:
          -   name: fix-permissions
              image: alpine:3.6
              command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
              securityContext:
                  privileged: true
              volumeMounts:
                  -   name: elasticsearch-data
                      mountPath: /usr/share/elasticsearch/data
          -   name: increase-vm-max-map
              image: alpine:3.6
              command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
              securityContext:
                  privileged: true
          -   name: fix-ulimit
              image: alpine:3.6
              command: ["sh", "-c", "ulimit -n 65536"]
              securityContext:
                  privileged: true
      hostname: elasticsearch
      restartPolicy: Always
      serviceAccountName: ""
      volumes:
      - name: elasticsearch-data
        persistentVolumeClaim:
          claimName: elasticsearch-data
      - name: elasticsearch-config
        configMap:
          name: elasticsearch-config

Can post configmap and pvc if needed but i think they are not really related to this issue.

score 0 · Answer 1 · answered May 23 '20 at 00:01

You probably can't fix it, it's very possible that Elasticsearch running in a container does not have the required system permissions.

mlockall is only required if you have swap enabled, which is uncommon on modern systems. If you don't have any swap space configured, don't worry about mlockall.

Elasticsearch with Kubernetes: Memory lock with bootstrap.memory_lock does not work

1 Answers1