Elastic Cloud, Filebeat, and Key/Token Authentication

Question

Is it possible configure filebeat to communicate with an Elastic Cloud instance using token authentication?

According to the docs, if I'm using a cloud instance I should configure cloud.id and cloud.auth in filebeat.yml

cloud.id: "..."
cloud.auth: "filebeat_setup:YOUR_PASSWORD"

The docs say that cloud.auth should be a username and password from my Elastic Cloud instance. I'd like to use an api_key instead. However, when I configure an API key

output.elasticsearch:
  # Authentication credentials - either API key or username/password.
  api_key: "key-id:key-value"

and attempt to test my connection,

$ sudo filebeat test output
elasticsearch: https://...:443...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 52.202.123.120, 18.214.74.184, 50.19.154.221
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... ERROR 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

It appears that filebeat doesn't acknowledge my API Key

"root_cause":[
    {
     "type":"security_exception",
     "reason":"missing authentication credentials for REST request [/]"
/* ... */

I've had success connecting to my cloud instance with the @elastic/elasticsearch javascript package using this token.

Before I continue to debug this probiem -- is it even possible to use token authentication to connection to Elasticsearch via filebeat? Or does filebeat only support username/password authentication?

score 1 · Accepted Answer · answered Nov 15 '20 at 18:39

The answer to this question turned out to be: Yes, you can use an api_key with filebeat, even if you're using elastic cloud.

While the error message received during my config test

missing authentication credentials for REST request

indicated the authentication was missing, the real problem was the key I had had previous success with had recently expired. I presume filebeat tried the API key, was rejected, and then fell back to trying the user credentials. When those credentials were missing, it gave the above error.

Elastic Cloud, Filebeat, and Key/Token Authentication

1 Answers1