1

Description

I want to query the data in our Elastic Cloud instance using the REST API with authorization via API Key.

Steps taken

I tried using the SQL API as well as the Search API. NB that I would much preferably use the SQL API.

Based on the following curl commands provided in the documentation:

curl -X POST "localhost:9200/_sql?format=txt&pretty" -H 'Content-Type: application/json' -d'
{
  "query": "SELECT * FROM library WHERE release_date < \u00272000-01-01\u0027"
}
'

[source]

curl -u elastic:password https://CLUSTER_ID.REGION.PLATFORM.found.io:9243/my_index/my_type -XPOST -d '{
"title": "One", "tags": ["ruby"]
}'
{"_index":"my_index","_type":"my_type","_id":"AV3ZeXsOMOVbmlCACuwj","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}

[source]

and the documentation about the REST API, I attempted the following:

import base64
import json
import requests

if __name__ == '__main__':
    response1 = requests.post(
        'https://{redacted-id}.northamerica-northeast1.gcp.elastic-cloud.com:9243/_sql?format=txt',
        headers={
            "Authorization": base64.standard_b64encode(bytes('{API_KEY_ID}:{API_KEY_KEY}', 'utf-8')),
            "Content-Type": 'application/json'
        },
        data={
            "query": """
                 ...
            """
        }
    )

    print('response1')
    print(response1)

    response2 = requests.get(
        'https://{redacted-id}.northamerica-northeast1.gcp.elastic-cloud.com:9243/logs-pubsub/_search',
        headers={
            "Authorization": base64.standard_b64encode(bytes('{API_KEY_ID}:{API_KEY_KEY}', 'utf-8')),
            "Content-Type": 'application/json'
        },
        data={
            "query": {
                 # ...
            }
        }
    )

    print('response2')
    print(response2)

But both queries answer with 404 - Not Found.

What did I miss? Am I missing a part of the path like /api/..., /rest/...? Or is this a misdirection from a 403 to a 404 and the issue is the API Key?

Thanks!

Philippe Hebert
  • 1,616
  • 2
  • 24
  • 51
  • Is using https://pypi.org/project/elasticsearch/ out of the scope ? – Paulo Feb 19 '22 at 22:17
  • We tend to limit our deps to a minimum. Since we only have one query to run and no plan to extend the integration at the moment, we'd prefer not to add an extra dependency. This being said, if you have a way to solve this using the elasticsearch client, I think this question will benefit. And if nothing else comes out I'll also consider that option. – Philippe Hebert Feb 20 '22 at 00:23

1 Answers1

6

The kind folks at ElasticStack provided me with the following answer. There were two issues:

The URL above was pointing to the Kibana instance, not to the ElasticCloud instance

In order to get the appropriate URL:

  1. Navigate to https://cloud.elastic.co/deployments/
  2. Click on your deployment:

Deployments overview

  1. Click on Copy Endpoint for the Elasticsearch endpoint:

Copy Endpoint

The Authorization header was malformed

The Authorization header for an ApiKey powered request is the following:

ApiKey {B64_ENCODE('API_KEY_ID:API_KEY_KEY')}

Which can be written in python as:

"ApiKey " + str(base64.standard_b64encode(bytes('API_KEY_ID:API_KEY_KEY', 'utf-8')), 'utf-8')

On a final note, the team also strongly suggested I look at their Free on-demand Elasticsearch training.

Hopefully this will be helpful to whoever passes by here!

Philippe Hebert
  • 1,616
  • 2
  • 24
  • 51