Podman with Elasticsearch Permissions Issues - Unable to access 'path.data' (/usr/share/elasticsearch/data)

Question

Now that RHEL8 is out Docker is no longer a viable option so I'm trying to get a cluster running in Podman, but have run into a permissions error. For whatever reason, no matter what I try, I get java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)

Directory setup

  mkdir /var/elasticsearch-data
  mkdir /var/elasticsearch-data/data01
  mkdir /var/elasticsearch-data/data02
  mkdir /var/elasticsearch-data/data03
  mkdir /var/elasticsearch-data/data04

  chmod -R g+rwx /var/elasticsearch-data
  chgrp -R 0 /var/elasticsearch-data

I eventually just gave up and set the directory with chmod -R 777 /var/elasticsearch-data and the error persists. I've tried changing the directory ownership to non-privileged users and I've tried changing the group.

Podman Compose File

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/elasticsearch-data/data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03,es04
      - cluster.initial_master_nodes=es01,es02,es03,es04
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/elasticsearch-data/data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02,es04
      - cluster.initial_master_nodes=es01,es02,es03,es04
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/elasticsearch-data/data03:/usr/share/elasticsearch/data
    networks:
      - elastic
  es04:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es04
    environment:
      - node.name=es04
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02,es03
      - cluster.initial_master_nodes=es01,es02,es03,es04
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/elasticsearch-data/data04:/usr/share/elasticsearch/data
    networks:
      - elastic
  kibana:
    image: docker.elastic.co/kibana/kibana:7.7.0
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200, http://es02:9200, http://es03:9200, http://es04:9200
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
  data04:
    driver: local

networks:
  elastic:
    driver: bridge

Run with podman-compose

podman-compose up

Log Output

[root@mapsdemo opt]# podman-compose up
podman pod create --name=opt --share net -p 9200:9200 -p 5601:5601
Error: unable to create pod: error adding pod to state: name opt is in use: pod already exists
125
podman create --name=es01 --pod=opt -l io.podman.compose.config-hash=123 -l io.podman.compose.project=opt -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=es01 -e node.name=es01 -e c
luster.name=es-docker-cluster -e discovery.seed_hosts=es02,es03 -e cluster.initial_master_nodes=es01,es02,es03 -e bootstrap.memory_lock=true -e ES_JAVA_OPTS=-Xms28g -Xmx28g --mount type=bind,source=/var/elasticsearch-data/data01,destinatio
n=/usr/share/elasticsearch/data --add-host es01:127.0.0.1 --add-host es01:127.0.0.1 --add-host es02:127.0.0.1 --add-host es02:127.0.0.1 --add-host es03:127.0.0.1 --add-host es03:127.0.0.1 --add-host es04:127.0.0.1 --add-host es03:127.0.0.1
 --add-host kibana:127.0.0.1 --add-host kib01:127.0.0.1 docker.elastic.co/elasticsearch/elasticsearch:7.7.0
Error: error creating container storage: the container name "es01" is already in use by "fc3902dad1e80d30d0cb778f2667356a26e2b005a7b24de032aad13ca7147622". You have to remove that container to be able to reuse that name.: that name is alre
ady in use
125
podman create --name=es02 --pod=opt -l io.podman.compose.config-hash=123 -l io.podman.compose.project=opt -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=es02 -e node.name=es02 -e c
luster.name=es-docker-cluster -e discovery.seed_hosts=es01,es03,es04 -e cluster.initial_master_nodes=es01,es02,es03,es04 -e bootstrap.memory_lock=true -e ES_JAVA_OPTS=-Xms28g -Xmx28g --mount type=bind,source=/var/elasticsearch-data/data02,
destination=/usr/share/elasticsearch/data --add-host es01:127.0.0.1 --add-host es01:127.0.0.1 --add-host es02:127.0.0.1 --add-host es02:127.0.0.1 --add-host es03:127.0.0.1 --add-host es03:127.0.0.1 --add-host es04:127.0.0.1 --add-host es03
:127.0.0.1 --add-host kibana:127.0.0.1 --add-host kib01:127.0.0.1 docker.elastic.co/elasticsearch/elasticsearch:7.7.0
Error: error creating container storage: the container name "es02" is already in use by "e66f93a50e61e62a1f0408aee3bef96f8ddf949f7edb35bfdbdf5557da91f743". You have to remove that container to be able to reuse that name.: that name is alre
ady in use
125
podman create --name=es03 --pod=opt -l io.podman.compose.config-hash=123 -l io.podman.compose.project=opt -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=es04 -e node.name=es04 -e c
luster.name=es-docker-cluster -e discovery.seed_hosts=es01,es02,es03 -e cluster.initial_master_nodes=es01,es02,es03,es04 -e bootstrap.memory_lock=true -e ES_JAVA_OPTS=-Xms28g -Xmx28g --mount type=bind,source=/var/elasticsearch-data/data04,
destination=/usr/share/elasticsearch/data --add-host es01:127.0.0.1 --add-host es01:127.0.0.1 --add-host es02:127.0.0.1 --add-host es02:127.0.0.1 --add-host es03:127.0.0.1 --add-host es03:127.0.0.1 --add-host es04:127.0.0.1 --add-host es03
:127.0.0.1 --add-host kibana:127.0.0.1 --add-host kib01:127.0.0.1 docker.elastic.co/elasticsearch/elasticsearch:7.7.0
Error: error creating container storage: the container name "es03" is already in use by "f707acc5bd286dcab77a5463b5d3ce04bceb55705203025926a87180b8ad8c2b". You have to remove that container to be able to reuse that name.: that name is alre
ady in use
125
podman create --name=kib01 --pod=opt -l io.podman.compose.config-hash=123 -l io.podman.compose.project=opt -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=kibana -e ELASTICSEARCH_UR
L=http://es01:9200 -e ELASTICSEARCH_HOSTS=http://es01:9200, http://es02:9200, http://es03:9200, http://es04:9200 --add-host es01:127.0.0.1 --add-host es01:127.0.0.1 --add-host es02:127.0.0.1 --add-host es02:127.0.0.1 --add-host es03:127.0.
0.1 --add-host es03:127.0.0.1 --add-host es04:127.0.0.1 --add-host es03:127.0.0.1 --add-host kibana:127.0.0.1 --add-host kib01:127.0.0.1 docker.elastic.co/kibana/kibana:7.7.0
Error: error creating container storage: the container name "kib01" is already in use by "3cc0578669211247b729d82cb4993c679164ebd5ea32858818b737bd6dcac293". You have to remove that container to be able to reuse that name.: that name is alr
eady in use
125
podman start -a es01
podman start -a es02
podman start -a es03
podman start -a kib01
{"type": "server", "timestamp": "2020-05-25T00:32:32,478Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "Unable to lock JVM Memory: error=12, reason=Cannot allocate
 memory" }
{"type": "server", "timestamp": "2020-05-25T00:32:32,481Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "This can result in part of the JVM being swapped out." }
{"type": "server", "timestamp": "2020-05-25T00:32:32,481Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 655
36" }
{"type": "server", "timestamp": "2020-05-25T00:32:32,481Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "These can be adjusted by modifying /etc/security/limits.co$
f, for example: \n\t# allow user 'elasticsearch' mlockall\n\telasticsearch soft memlock unlimited\n\telasticsearch hard memlock unlimited" }
{"type": "server", "timestamp": "2020-05-25T00:32:32,482Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "If you are logged in interactively, you will have to re-log
in for the new limits to take effect." }
{"type": "server", "timestamp": "2020-05-25T00:32:32,537Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "uncaught exception in thread [m
ain]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)",

Has anyone else run into this?

`Unable to access 'path.data` this is the main cause of error, make sure there isn't any other Elasticsearch process is running and user who is running the new elasticsearch process has write access to it. — Amit, May 25 '20 at 00:47
Negative. This is a completely clean box. That `podman-compose` is the only thing going on. No other containers running outside what is listed here. This is one of those things I've done a million times with Docker so I'm either brain farting (possible) or this is a Podman specific problem. — Grant Curell, May 25 '20 at 00:51

score 1 · Accepted Answer · edited Feb 14 '21 at 14:35

1

if you run the following:

chcon -t svirt_sandbox_file_t <host file path>

This should enable you to use selinux (which is a good idea)

edited Feb 14 '21 at 14:35

John Conde

217,595
99
455
496

answered Feb 14 '21 at 13:45

James

26
1

score 0 · Answer 2 · answered May 25 '20 at 20:05

This is not a real answer to the question, but if it helps anyone else who encounters this, this is how to install Docker on RHEL-8. You have to force install a more up to date version of containerd. I wasn't able to figure out what was wrong with Podman compose's permissions.

  dnf install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
  dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
  curl -L "https://github.com/docker/compose/releases/download/1.25.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  systemctl enable docker
  systemctl start docker

Grant Curell · Answer 3 · 2021-02-14T20:22:57.660

Edit: Use @James' answer - using SELinux is always a better idea. I didn't bother figuring it out for my lab setup, but since someone else has you should do that.

-- Original Answer --

I think it's probably strange to answer my own post twice, but I hope this is handy.

Are you ready for the surprise of a lifetime? Are you ready?

I'll even give you a hint - I'm running on RHEL8 vanilla.

I know this is going to be a shocker.

pause for dramatic effect

SELinux.

Turn it off (setenforce 0). Everything magically works.

I can hear how shocked everyone is from here.

Podman with Elasticsearch Permissions Issues - Unable to access 'path.data' (/usr/share/elasticsearch/data)

Directory setup

Podman Compose File

Run with podman-compose

Log Output

3 Answers3