0

I am trying to setup the Elastic Stack using Docker Compose. For background, I am running on a Windows 10 device with WSL2 and Docker Desktop (Docker Compose version v2.20.2-desktop.1) installed. I have changed the virtual memory configuration using the sysctl -w vm.max_map_count=262144 command.

I have created two files in the directory C:\Project Root\esk. The first is my docker-compose.yaml file, and the other is my .env file. When I run the docker-compose up -d command, I get errors for every environment variable saying it is not set. Here are the errors:

time="2023-08-14T23:04:47+10:00" level=warning msg="The \"STACK_VERSION\" variable is not set. Defaulting to a blank string."
time="2023-08-14T23:04:47+10:00" level=warning msg="The \"KIBANA_PORT\" variable is not set. Defaulting to a blank string."
time="2023-08-14T23:04:47+10:00" level=warning msg="The \"KIBANA_PASSWORD\" variable is not set. Defaulting to a blank string."
time="2023-08-14T23:04:47+10:00" level=warning msg="The \"XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY\" variable is not set. Defaulting to a blank string."
time="2023-08-14T23:04:47+10:00" level=warning msg="The \"XPACK_SECURITY_ENCRYPTIONKEY\" variable is not set. Defaulting to a blank string."
time="2023-08-14T23:04:47+10:00" level=warning msg="The \"XPACK_REPORTING_ENCRYPTIONKEY\" variable is not set. Defaulting to a blank string."
time="2023-08-14T23:04:47+10:00" level=warning msg="The \"MEM_LIMIT\" variable is not set. Defaulting to a blank string."
parsing C:\root\esk\docker-compose.yaml: error while interpolating services.kibana.mem_limit: failed to cast to expected type: invalid size: ''

Here is my .env file:

ELASTIC_PASSWORD=password
KIBANA_PASSWORD=password
STACK_VERSION=8.4.3
CLUSTER_NAME=docker-cluster
LICENSE=basic
ES_PORT=9200
KIBANA_PORT=5601
MEM_LIMIT=1073741824
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=A54iKDSEyaum2vI4wJRm6RYJwYFS6Ov1
XPACK_SECURITY_ENCRYPTIONKEY=wR4Wbetgv9sostKvoOGPj0stwUMqeaDa
XPACK_REPORTING_ENCRYPTIONKEY=9h72FEdcxf0OvDsmkCZzhl5psPT1hoj1

And here is my docker-compose.yaml:

version: "3.9"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - ../certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - ../certs:/usr/share/elasticsearch/config/certs
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02
      - discovery.seed_hosts=es02
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - ../certs:/usr/share/elasticsearch/config/certs
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02
      - discovery.seed_hosts=es01
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
  kibana:
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - ../certs:/usr/share/kibana/config/certs
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY}
      - XPACK_SECURITY_ENCRYPTIONKEY=${XPACK_SECURITY_ENCRYPTIONKEY}
      - XPACK_REPORTING_ENCRYPTIONKEY=${XPACK_REPORTING_ENCRYPTIONKEY}
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
networks:
  default:
    name: elastic-stack-network
volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  kibanadata:
    driver: local

I've tried removing line breaks, quoting the variables, rebooting machine, reinstalling docker, running as admin, changing my system locale (I was originally Japanese locale), but nothing makes a difference. Sometimes when I try run it - even without making changes to the .env file - it will only give the error for some of the variables and not others. It's seemingly arbitrary.

punished ksen
  • 1
  • 1
  • I copied both your .env and .yml files and ran `docker compose convert` locally and all environment variables were replaced exactly as expected. `docker compose up -d` ran exactly as expected, all services started just fine. So there's something off with your particular environment. Are you sure the .env file is in the same folder as the .yml file? – Mr. T Aug 14 '23 at 14:50
  • Hi, thanks for looking into it. Yeah, both are in the same folder. What version of WSL2 and Docker are you running? I may try a different version or just try it on a different device all together. – punished ksen Aug 15 '23 at 07:59
  • Have you tried explicitly referencing the files? `docker compose --env-file [env-file].env -f [docker-compose-file].yml` Other than that: may be a line break issue within the env-file? I mean are you sure the file contains windows line breaks? – Kekzpanda Aug 15 '23 at 14:55
  • Okay, I re-wrote the files all from scratch to account for any line break issues. I also tried explicitly referencing the files and it works. Thank you very much. – punished ksen Aug 16 '23 at 10:56
  • I notice that the command you specified for your `setup` service refers to environment variables, but you haven't included an `environment` section for that service. I think you need to explicitly import the environment variables by including `- MY_VAR=${MY_VAR}` elements (as you have done for the `ELASTIC_PASSWORD` variable for the `es01` service) – Jeff Sep 01 '23 at 17:51

0 Answers0