How to send data into Elastic Cloud from Java?

Question

I want to insert (index) some data into Elastic Search running in Elastic Cloud in a Java application.

To do so, I wrote the following piece of code:

void sendStuffToElasticSearch() {
    RestHighLevelClient client = null;
    try {
        client = new RestHighLevelClient(
                RestClient.builder(CLOUD_ID)
        );

        RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
        builder.addHeader("Authorization", String.format("ApiKey %s",
                API_KEY));
        final RequestOptions requestOptions = builder.build();

        IndexRequest request = new IndexRequest("posts");
        request.id("1");
        String jsonString = "{" +
                "\"user\":\"kimchy\"," +
                "\"postDate\":\"2013-01-30\"," +
                "\"message\":\"trying out Elasticsearch\"" +
                "}";
        request.source(jsonString, XContentType.JSON);
        IndexResponse indexResponse = client.index(request, requestOptions);
        System.out.println("indexResponse");
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        IOUtils.closeQuietly(client);
    }
}

API_KEY is the key I generated according to this tutorial which also says I need to send it in the Authorization header in the following format: Authorization: ApiKey $EC_API_KEY.

When I run the above code, I am getting the following error:

org.elasticsearch.client.ResponseException: method [PUT], host [https://XXXXXXXXXX:9243], URI [/posts/_doc/1?timeout=1m], status line [HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/posts/_doc/1?timeout=1m]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/posts/_doc/1?timeout=1m]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1621)
        ... 30 more

How can fix this, i. e. provide all authentication-related data in the way Elastic Cloud expects them?

I am using following libraries:

    <properties>
        [...]
        <elastic-search-client.version>7.11.1</elastic-search-client.version>
    </properties>

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>${elastic-search-client.version}</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>${elastic-search-client.version}</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>${elastic-search-client.version}</version>
        </dependency>

Update 1: Base64-encoding of the API key as suggested here (see code below) did not help.

            RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
            builder.addHeader("Authorization", String.format("ApiKey %s",
                    Base64.getEncoder().encodeToString(API_KEY.getBytes(StandardCharsets.UTF_8))));
            final RequestOptions requestOptions = builder.build();

Update 2: Changing the way I create the client, did not help, either (see below).

            Header[] defaultHeaders =
                    new Header[]{new BasicHeader("Authorization",
                            String.format("ApiKey %s",API_KEY))};
            final RestClientBuilder builder1 = RestClient.builder(CLOUD_ID);
            builder1.setDefaultHeaders(defaultHeaders);
            client = new RestHighLevelClient(
                    builder1
            );

Update 3: I changed the supplied API key to

public static final String BASE64_API_KEY = Base64.getEncoder().encodeToString(String.format("%s:%s", ID, KEY).getBytes());

as suggested by Ricardo Ferreira.

Now I am getting a different error:

org.elasticsearch.client.ResponseException: method [PUT], host [XXXXXXXXXXXXXXXX], URI [/posts/_doc/1?timeout=1m], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [XXXXXXXXXXXXXXXX] of user [XXXXXXXXXXXXXXXX]"}],"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [XXXXXXXXXXXXXXXX] of user [XXXXXXXXXXXXXXXX]"},"status":403}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1621)
        ... 30 more

Update 4:

After I created the index in question, the error message changed to this:

org.elasticsearch.client.ResponseException: method [PUT], host [XXXXXXXXXXXXXXXX], URI [/camunda-1/_doc/1?timeout=1m], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/bulk[s]] is unauthorized for API key id [XXXXXXXXXXXXXXXX] of user [XXXXXXXXXXXXXXXX]"}],"type":"security_exception","reason":"action [indices:data/write/bulk[s]] is unauthorized for API key id [XXXXXXXXXXXXXXXX] of user [XXXXXXXXXXXXXXXX]"},"status":403}

score 3 · Accepted Answer · edited Jan 13 '23 at 13:07

It's not working because you are using the wrong API Key.

But don't worry: these things happen a lot. It certainly happened to me.

The API Key that you are creating is for you to issue REST requests against Elasticsearch Service — which is the entity that governs your Elasticsearch and Kibana clusters.

To make it work, you need to create an API Key from Elasticsearch specifically. To create one, go to the Dev Tools Console and issue the following request:

POST _security/api_key
{
   "name": "my-api-key",
   "expiration": "7d", 
   "role_descriptors": {
      "custom-role": {
         "cluster": ["all"],
         "index": [
            {
               "names": [
                  "index-1",
                  "index-2"
               ],
               "privileges": ["all"]
            }
         ]
      }
   }
}

If executed successfully, you will get a response like this:

{
  "id" : "liKs_XcBrNsSAgwboCN9",
  "name" : "my-api-key",
  "expiration" : 1615473484899,
  "api_key" : "NC3ZeIb_SGWjGJRZVoOf2g"
}

Take note of the fields id and api_key. You are going to need them to create the authorization header:

String apiKey = String.format("%s:%s", id, api_key);
apiKey = Base64.getEncoder().encodeToString(apiKey.getBytes());
String authorization = String.format("ApiKey %s", apiKey);

After this just use the authorization in your Java code:

builder.addHeader("Authorization", authorization);

How to send data into Elastic Cloud from Java?

1 Answers1