Elasticsearch - Authentication error when check to see if index exist

Question

I have my elastic search with cognito authentication enabled for kibana. This is working fine as expected.

In my python script I connect to elasticsearch by providing username/password in http_auth(), while creating the connection object. But when I attempt to check if an indices exist, am getting authentication error? Can someone please help. Heres the sample piece of code for your simulation please.

from __future__ import print_function
import json
import time
import urllib
import re
import sys
import requests
import base64
import time
from elasticsearch import Elasticsearch
from datetime import datetime

esEndpoint = ""
uname
pwd
indexName = 'index_1'

mappings_rds = {
    "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
    },
    "mappings": {
      "properties" : {
         "tableName": {
            "type": "keyword"
          },
         "tableRows": {
            "type": "integer"
          },        
         "updatedTime": {
            "type": "date",
            "format":"date_optional_time||yyyy-MM-dd'T'HH:mm:ss"
          },
        "created_timestamp":{
            "type": "date",
            "format":"date_optional_time||yyyy-MM-dd'T'HH:mm:ss"
          }
      }
    }
}

esClient = Elasticsearch([esEndPoint],http_auth=(uname,pwd))

try:
      res = esClient.indices.exists(indexName)
      print(res)
      if res is False:
          r = esClient.indices.create(indexName, body=mapping_rds, ignore=400)
      return 1
  except Exception as E:
          print("Unable to Create Index {0}".format(indexName))

Access kibana using (uname,pwd) and let us know what do you see when you run `HEAD /index_1` — Assael Azran, Oct 19 '20 at 09:42
thanks Assael, I tried the command and getting "{"statusCode":404,"error":"Not Found","message":"404 - Not Found"}" as resonse in the kibana console. — Yuva, Oct 19 '20 at 10:06

score 0 · Accepted Answer · answered Oct 21 '20 at 07:18

Thank you all for your comments. I had a call with AWS Support team regarding this. The issue was found to be the username & password that I had used. I was using the user credentials created using AWS Cognito. The AWS support team confirmed that the said credentials is only meant for Kibana access, and it cannot be used for connecting to Elasticsearch from databricks / python script.

More info available here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html

Once I generate the access/secret keys, and use it from my python script, am able to connect to elasticsearch to create index.

Sharing a sample code for future reference:

session=boto3.session.Session(aws_access_key_id=akey, aws_secret_access_key=skey, region_name=region)
credentials = session.get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
print(credentials.access_key, credentials.secret_key, credentials.token)
es = Elasticsearch(
    hosts = [{'host': esEndpoint, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)

Thanks

Elasticsearch - Authentication error when check to see if index exist

1 Answers1