10

I am trying to use nfs mount as my elasticsearch data directory, earlier I was using host storage. But when I am trying to up my container, I am facing below mentioned error :

 ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes/0];
es02    | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0
es02    |   at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
es02    |   at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
es02    |   at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
es02    |   at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
es02    |   at java.base/java.nio.file.Files.createDirectory(Files.java:694)
es02    |   at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:801)
es02    |   at java.base/java.nio.file.Files.createDirectories(Files.java:787)
es02    |   at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:275)
es02    |   at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212)
es02    |   at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:272)
es02    |   at org.elasticsearch.node.Node.<init>(Node.java:362)
es02    |   at org.elasticsearch.node.Node.<init>(Node.java:289)

I am using docker-compose for the same earlier my cluster was working fine,when i was using host storage but when i tried to switch to NFS storage i am facing above mentioned error. Below is my docker-compose file:

version: '3.3'
volumes:
  data:
    driver: local
    driver_opts:
       type: nfs
       device: ":/data1/elasticsearch_data"
       o: addr=10.10.15.46,rw
#volumes:
#  repo:
#    driver: local
#    driver_opts:
#       type: none
#       device: /apm_backup
#       o: bind
#
services:
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    container_name: es02
    volumes:
      - data:/usr/share/elasticsearch/data
#      - ./unicast_hosts.txt:/usr/share/elasticsearch/config/unicast_hosts.txt
    environment:
      - node.name=es02
      - node.master=true
      - node.data=true
      - cluster.name=apm-docker-cluster
      - discovery.seed_hosts=es01
#      - discovery.seed_providers=file
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms10g -Xmx10g"
      - XPACK_SECURITY_ENABLED=false
#      - path.repo=/apm_backup
#    command: ["elasticsearch", "-Elogger.level=DEBUG"]
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    restart: "unless-stopped"
    logging:
      driver: 'json-file'
      options:
          max-size: '2m'
          max-file: '5'
    ports:
      - 9200:9200
      - 9300:9300
    extra_hosts:
      - "es01:10.10.26.27"
      - "es02:10.10.26.111"
      - "es03:10.10.26.111"
    network_mode: "host"
    labels:
      org.label-schema.group: "APM-monitoring"
    healthcheck:
      test: ["CMD", "curl", "--write-out", "'HTTP %{http_code}'", "--silent", "--output", "/dev/null", "http://es02:9200/"]
      retries: 10
      interval: 10s

This is my directory permission and location

What I have tried till now:

chown 1000:1000 elasticsearch_data (1000 uid of vimzy user mentioned in photo).

I have tried priviliged=true in docker-compose. and apart from this tried multiple ways to counter permission denied issue.

Any help...

YLR
  • 1,503
  • 4
  • 21
  • 28
vansh madan
  • 128
  • 1
  • 1
  • 10

4 Answers4

14

This error is occurring due to permission issue on specified path.

Change the path from /usr/share/elasticsearch/data to /var/lib/elasticsearch/data

It will work!

Sagar Kulthe
  • 516
  • 3
  • 14
  • what is the permission issue that you mentioned? – ORHAN ERDAY Oct 12 '22 at 07:33
  • 2
    For elastic version 7 use the /var/lib/elasticsearch/data. For elastic version 8 is /usr/share/elasticsearch/data – C.LS Nov 24 '22 at 16:13
  • 6
    will start, but won't do what's intended. Will not write data over the mounted volume, instead will write to internal container path "/usr/share/elasticsearch/data", which is the correct path even for v7. – MatteoSp Dec 16 '22 at 12:20
5

this fixed the problem for me

sudo chown -R 1000:root docker_data
FreezeSoul
  • 51
  • 1
  • 2
0

It might be an incompatibility with NFSv3, try to switch to NFSv4 as detailed in this blog:

https://www.frakkingsweet.com/elasticsearch-nfs-and-locking-issues/

Ricardo Ferreira
  • 1,236
  • 3
  • 6
  • ` o: "addr=10.10.15.46,vers=4,rw" ` Hi added this as well but no changes same error as mentioned above @RIcardo – vansh madan Jan 11 '21 at 17:24
  • I think you have to enable v4 in the NFS server as well. Remember that updating the volume options and redeploying the stack does not modify an already existing volume. Volumes also don't get removed from the docker swarm nodes when they are no longer in use. You need to go to each individual node and manually delete the volumes using docker volume rm . – Ricardo Ferreira Jan 11 '21 at 17:34
  • Hi Ricardo, I am using docker-compose down -v to remove volumes orphanges volumes,containers . And V4 is already enable on NFS server – vansh madan Jan 12 '21 at 05:40
0

The following solution worked for me:

I formatted my NFS Disk and then mounted the nfs disk as a device in docker-compose.yml

version: '3.3'
volumes:
  data:
    driver: local
    driver_opts:
       type: nfs
       device: ":/data1/elasticsearch_data"
       o: addr=10.10.15.46,rw
vansh madan
  • 128
  • 1
  • 1
  • 10
  • 1
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Dec 12 '21 at 01:06