Metricbeat Beat failed to connect es (xpack enabled)

Question

Helm Version: "v3.1.1"

helm get release output

helm status metricbeat NAME: metricbeat LAST DEPLOYED: Thu Apr 30 10:05:32 2020 NAMESPACE: default STATUS: deployed REVISION: 1

Describe the bug: Unable to connect to elasticsearch, deployed configmap does not contain username: and password input.

Steps to reproduce:

1. Enable xpack in elasticsearch
2. run - helm install metricbeat elastic/metricbeat --set imageTag=7.6.2 --values metrics.yaml

metrics.yaml

daemonset:
  extraEnvs:
    - name: 'ES_USERNAME'
      valueFrom:
        secretKeyRef:
          name: elastic-credentials
          key: username
    - name: 'ES_PASSWORD'
      valueFrom:
        secretKeyRef:
          name: elastic-credentials
          key: password
  # Allows you to add any config files in /usr/share/metricbeat
  # such as metricbeat.yml for daemonset
  metricbeatConfig:
    metricbeat.yml: |
      metricbeat.modules:
      - module: kubernetes
        metricsets:
          - container
          - node
          - pod
          - system
          - volume
        period: 10s
        host: "${NODE_NAME}"
        hosts: ["${NODE_NAME}:10250"]
        # bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        # ssl.verification_mode: "none"
        # If using Red Hat OpenShift remove ssl.verification_mode entry and
        # uncomment these settings:
        #ssl.certificate_authorities:
          #- /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        processors:
        - add_kubernetes_metadata: ~
      - module: kubernetes
        enabled: true
        metricsets:
          - event
      - module: system
        period: 10s
        metricsets:
          - cpu
          - load
          - memory
          - network
          - process
          - process_summary
        processes: ['.*']
        process.include_top_n:
          by_cpu: 5
          by_memory: 5
      - module: system
        period: 1m
        metricsets:
          - filesystem
          - fsstat
        processors:
        - drop_event.when.regexp:
            system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
      output.elasticsearch:
        username: '${ES_USERNAME}'
        password: '${ES_PASSWORD}'
        hosts: ["elasticsearch-master:9200"]


deployment:
  extraEnvs:
    - name: 'ES_USERNAME'
      valueFrom:
        secretKeyRef:
          name: elastic-credentials
          key: username
    - name: ES_PASSWORD'
      valueFrom:
        secretKeyRef:
          name: elastic-credentials
          key: password
  # Allows you to add any config files in /usr/share/metricbeat
  # such as metricbeat.yml for deployment
  metricbeatConfig:
    metricbeat.yml: |
      metricbeat.modules:
      - module: kubernetes
        enabled: true
        metricsets:
          - state_node
          - state_deployment
          - state_replicaset
          - state_pod
          - state_container
        period: 10s
        hosts: ["${KUBE_STATE_METRICS_HOSTS}"]
      output.elasticsearch:
        username: '${ES_USERNAME}'
        password: '${ES_PASSWORD}'
        hosts: ["elasticsearch-master:9200"]

metricbeat.yml after deployment does not contain username and password inputs:

 metricbeat.modules:
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      host: "${NODE_NAME}"
      hosts: ["${NODE_NAME}:10255"]
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
    - module: kubernetes
      enabled: true
      metricsets:
        - event
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5
        by_memory: 5
    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'

Expected behavior: should connect to es using username and password without an issue.

Provide logs and/or server output (if relevant):

   pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://elasticsearch-master:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

Any additional context:

I am able to connect elastic search using same credentials and it works fine with Kibana login as well.

Answer 1

I'm the maintainer of Elastic Helm charts.

Due to a recent refactoring, the values daemonset.xxx and deployment.xxx documented on Git repository master branch aren't compatible with the current released chart version.

If you want to install the current released version (7.6.2), you can find the values to use on the README from 7.6.2 release.

If you want to deploy the next version which should be released soon and will use daemonset.xxx and deployment.xxx values, you can clone the repo and use the installation instructions from 7.7 branch.

PS: As mentioned in comment, please note that Elastic charts don't support Helm v3 yet, we still advise using Helm v2.