Elasticsearch with xpack security fails

Question

I am trying to set up a simple ELK stack using docker. While I disable xpack security it starts fine and I can access the Kibana interface. If xpack security is enabled I get an "Kibana server is not ready yet" error from the Kibana interface. This error is most likely caused by this Elasticsearch error:

{"type": "server", "timestamp": "2020-08-03T15:35:10,134Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elastic-cluster", "node.name": "elasticsearch", "message": "Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.monitoring-es-7-2020.08.03][0]]]).", "cluster.uuid": "Vdk1-_4sSvuqlEspQcF-6A", "node.id": "PZMUpi_JSJS6IZ7tv6H22g"  }
{"type": "server", "timestamp": "2020-08-03T15:35:10,560Z", "level": "ERROR", "component": "o.e.x.s.a.e.NativeUsersStore", "cluster.name": "elastic-cluster", "node.name": "elasticsearch", "message": "security index is unavailable. short circuiting retrieval of user [elasticadmin]", "cluster.uuid": "Vdk1-_4sSvuqlEspQcF-6A", "node.id": "PZMUpi_JSJS6IZ7tv6H22g"  }

This is my elasticsearch.yml:

cluster.name: elastic-cluster
node.name:    elasticsearch
network.host: 0.0.0.0
transport.host: 0.0.0.0

## Cluster Settings
discovery.seed_hosts: elasticsearch
cluster.initial_master_nodes: elasticsearch

## License
xpack.license.self_generated.type: basic

# Security
xpack.security.enabled: true

## - ssl
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/elasticsearch.key
xpack.security.transport.ssl.certificate: certs/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt

## - http
#xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.key: certs/elasticsearch.key
#xpack.security.http.ssl.certificate: certs/elasticsearch.crt
#xpack.security.http.ssl.certificate_authorities: certs/ca.crt
#xpack.security.http.ssl.client_authentication: optional

# Monitoring
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true

This is the error log from Kibana:

{"type":"log","@timestamp":"2020-08-03T15:42:22Z","tags":["warning","plugins","licensing"],"pid":6,"
message":"License information could not be obtained from Elasticsearch due to [security_exception] unable to authenticate user [elasticadmin] for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elasticadmin] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [elasticadmin] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}

Basic curl request:

curl -H "Authorization: Basic ZWxhc3RpY2FkbWluOjEyMzQ1Njc4OQ==" -XGET "http://localhost:9200/_cat/nodes?v&pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [elasticadmin] for REST request [/_cat/nodes?v&pretty]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [elasticadmin] for REST request [/_cat/nodes?v&pretty]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
}

Another Auth request:

docker@docker:~$ curl -H "Authorization: Basic ZWxhc3RpY2FkbWluOjEyMzQ1Njc4OQ" -XGET "http://localhost:9200/_security/_authenticate"
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elasticadmin] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"unable to authenticate user [elasticadmin] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

Docker-Compose:

secrets:
  elasticsearch.keystore:
    file: ${ELK_DATA}/secrets/keystore/elasticsearch.keystore
  elastic.ca:
    file: ${ELK_DATA}/secrets/certs/ca/ca.crt
  elasticsearch.certificate:
    file: ${ELK_DATA}/secrets/certs/elasticsearch/elasticsearch.crt
  elasticsearch.key:
    file: ${ELK_DATA}/secrets/certs/elasticsearch/elasticsearch.key
  kibana.certificate:
    file: ${ELK_DATA}/secrets/certs/kibana/kibana.crt
  kibana.key:
    file: ${ELK_DATA}/secrets/certs/kibana/kibana.key

services:

####################################################################
############################# ELK ##################################
####################################################################

  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
    restart: unless-stopped
    environment:
      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      ELASTIC_CLUSTER_NAME: ${ELASTIC_CLUSTER_NAME}
      ELASTIC_NODE_NAME: ${ELASTIC_NODE_NAME}
      ELASTIC_INIT_MASTER_NODE: ${ELASTIC_INIT_MASTER_NODE}
      ELASTIC_DISCOVERY_SEEDS: ${ELASTIC_DISCOVERY_SEEDS}
      ES_JAVA_OPTS: -Xmx${ELASTICSEARCH_HEAP} -Xms${ELASTICSEARCH_HEAP} -Des.enforce.bootstrap.checks=true
      bootstrap.memory_lock: "true"
    volumes:
      - ${ELK_DATA}/elasticsearch/data:/usr/share/elasticsearch/data
      - ${ELK_DATA}/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ${ELK_DATA}/elasticsearch/config/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties
    secrets:
      - source: elasticsearch.keystore
        target: /usr/share/elasticsearch/config/elasticsearch.keystore
      - source: elastic.ca
        target: /usr/share/elasticsearch/config/certs/ca.crt
      - source: elasticsearch.certificate
        target: /usr/share/elasticsearch/config/certs/elasticsearch.crt
      - source: elasticsearch.key
        target: /usr/share/elasticsearch/config/certs/elasticsearch.key
    ports:
      - 9200:9200
      - 9300:9300
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 200000
        hard: 200000
    networks:
      - traefik_proxy
      
  logstash:
    container_name: logstash
    image: docker.elastic.co/logstash/logstash:${ELK_VERSION}
    restart: unless-stopped
    volumes:
      - ${ELK_DATA}/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - ${ELK_DATA}/logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
      - ${ELK_DATA}/logstash/pipeline:/usr/share/logstash/pipeline
    environment:
      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      ELASTICSEARCH_HOST_PORT: ${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}
      LS_JAVA_OPTS: "-Xmx${LOGSTASH_HEAP} -Xms${LOGSTASH_HEAP}"
    ports:
      - 5044:5044
      - 9600:9600
    networks:
      - traefik_proxy

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:${ELK_VERSION}
    restart: unless-stopped
    volumes:
      - ${ELK_DATA}/kibana/config:/usr/share/kibana/config
    environment:
      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      ELASTICSEARCH_HOST_PORT: ${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}
    secrets:
      - source: elastic.ca
        target: /certs/ca.crt
      - source: kibana.certificate
        target: /certs/kibana.crt
      - source: kibana.key
        target: /certs/kibana.key
    ports:
      - 5601:5601
    networks:
      - traefik_proxy

Where should I start looking to find the source of this issue?

Thanks for any help!

score 0 · Answer 1 · answered Aug 04 '20 at 07:32

0

when you enable x-pack, elasticsearch is getting started, But it seems your kibana is not getting authenicated.please see below part of your error message which explains this.

elasticadmin user is not authenticated

Please see this user and see you are passing the correction authentication while accessing elasticsearch. You need to pass username and password under basic authentication mechanism.

answered Aug 04 '20 at 07:32

Amit

30,756
6
57
88

Thanks for your comment. I've tried to send a request to elasticsearch using the user and password and I'm getting the same error. Isn't this issue on the elasticsearch side? – In0cenT Aug 04 '20 at 07:52
@In0cenT r u sending correct creds for user `elasticadmin` ? please use https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-authenticate.html to verify that – Amit Aug 04 '20 at 07:54
I've added my docker-compose how I've set it up. In my .env I currently have the user set to elasticadmin and password 123456789. I then used this site to get the Base64 for the Authentication token: https://www.base64encode.org/ I could not authenticate following the guide as you can see in the logs in the original post. – In0cenT Aug 04 '20 at 08:04
Is elastic the only valid user? I've change my username in my .env to elastic and it worked perfectly fine. After changing it back to an other username it stopped working until I've set it back to elastic. I have re-generated all keys after changing the password and/or user. – In0cenT Aug 04 '20 at 09:09
Your answer did not help my issue. The issue was that I tried to set the container up with an unsupported username. – In0cenT Aug 04 '20 at 11:38

score 0 · Answer 2 · answered Feb 08 '21 at 15:42

I have the same issue but I solve it:

1 Step

you can configure you docker compose as

kibana:    
build: kibana    
container_name: kibana       
ports:
  - 5601:5601
volumes:
  - ./kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
networks:
  backend:
    aliases:
      - "kibana"

2 Step

and my kibana file is that:

...
elasticsearch.username: "kibana"
elasticsearch.password: "mypwd"
...

and my Dockerfile is:

FROM docker.elastic.co/kibana/kibana:7.10.2 
COPY kibana.yml /usr/share/kibana/kibana.yml
USER root
RUN chown root:kibana /usr/share/kibana/config/kibana.yml
USER kibana

score 0 · Answer 3 · answered Apr 20 '21 at 12:09

I got this issue when the data folder of ElasticSearch was deleted and re-initialized from scratch afterwards. The point is that the built-in users were not initialized.

As soon as I initialized the built-in users the error disappeared and the system worked again.

bin/elasticsearch-setup-passwords interactive|auto [-u "https://<host_name>:9200"]

Elasticsearch with xpack security fails

3 Answers3