Authntication finally failed error on AWS ElasticSearch Service

Question

I'm trying to upgrade a Django application from using AWS ElasticSearch 2.3 to 7.4 (and upgrade the Django packages at the same time).

I have it running locally, but when I attempt to run with AWS ElasticSearch7.4 I get the following Traceback

Traceback (most recent call last):
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/henry/Documents/Sites/Development/autumna2/autumna/src/search/views.py", line 169, in name_suggestions
    field='name_suggestion',
  File "/home/henry/Documents/Sites/Development/autumna2/autumna/src/search/views.py", line 129, in search_suggestions
    data = sqs.execute(ignore_cache=True).to_dict()['hits']['hits']
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/elasticsearch_dsl/search.py", line 698, in execute
    **self._params
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 1627, in search
    body=body,
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/elasticsearch/transport.py", line 362, in perform_request
    timeout=timeout,
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 248, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/henry/Documents/Sites/Development/autumna2/env/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 244, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, 'Authentication finally failed')

I am using django-elasticsearch-dsl so I've simply declared the host within settings.py (this is a straight replacement of what I had before) with

ELASTICSEARCH_DSL = {
    'default': {
        'hosts': 'https://search-elastic7-zoqohjbiedz2ozmthfi4a3ccm4.eu-west-2.es.amazonaws.com', # I've changed this string
    },
}

I'm using IAM authentication and I've given the IAM account full access to all my ElasticSearch instances (but this is different to the Master user and password I used when setting up the instance). If I need those, how do I pass them? If not, what am I doing wrong?

score 1 · Accepted Answer · answered Mar 25 '20 at 12:43

The following sorted it if anyone else has an issue

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID',"some key")
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY',"another secret key")
AWS_SERVICE = 'es'
AWS_ELASTICSEARCH = os.environ.get('AWS_ELASTICSEARCH','search-elastic7-zoqohjbiedz2ozmthfi4a3ccm4.eu-west-2.es.amazonaws.com')
http_auth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AWS_SERVICE)
ELASTICSEARCH_DSL = {
    'default': {
        'hosts': [{'host': AWS_ELASTICSEARCH, 'port': 443}],
        'http_auth' : http_auth,
        'use_ssl' : True,
        'verify_certs' : True,
        'connection_class' : RequestsHttpConnection
    },

score 0 · Answer 2 · answered Mar 25 '20 at 12:36

It doesn't look like django-elasticsearch-dsl supports AWS ES request signing which is required when you have IAM auth enabled: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html

You can check if that's the problem by making the AWS ES instance completely open (no auth).

Authntication finally failed error on AWS ElasticSearch Service

2 Answers2