AuthorizationException: The security token included in the request is expired

Question

I have an AWS Fargate that needs to query Elasticsearch and Dynamodb. The role associated with the cluster has permission to access those services. After a while (about 40 minutes) I start to get this error AuthorizationException: TransportError(403, '{"message":"The security token included in the request is expired"}'). It's the error raised when I try to access Elasticsearch and if I try to access Dynamodb, I don't get that error. I'm using boto3 version 1.9.160 and I get the credentials using these lines of codes:

session = boto3.Session()
dynamodb_client_nvirginia = session.client(service_name='dynamodb')
aws_auth = AWS4Auth(
  session.get_credentials().access_key, session.get_credentials().secret_key, 'us-east-1', 'es', session_token=session.get_credentials().token)
elasticsearch_client = Elasticsearch(
  hosts=[{'host': 'my-elasticsearch-host', 'port': 443}],
  http_auth=aws_auth, use_ssl=True, verify_certs=True,
  connection_class=RequestsHttpConnection, timeout=30, max_retries=10, retry_on_timeout=True)

I read that the credentials are refreshed automatically by boto3.

As a try, I decided to refresh the credentials to connect to Elasticsearch after 30 minutes but I'm still getting the same error.

What am I doing wrong?

score 0 · Answer 1 · answered Nov 26 '19 at 12:26

I decided to use a Singleton to manage this error and refresh client credentials. I created this class:

class ElasticsearchClientInstanceGenerator:

    __instance = None

    def __init__(self):
        """ Virtually private constructor. """
        if self.__instance is not None:
            raise Exception("This class is a singleton!")
        else:
            ElasticsearchClientInstanceGenerator.__instance = self
        self.__elasticsearch_client = None
        self.__host: Optional[str] = None
        self.__port: Optional[int] = None
        self.__use_ssl: Optional[bool] = None
        self.__verify_certs: Optional[bool] = None

    def generate_elasticsearch_client(self):
        import boto3
        from requests_aws4auth import AWS4Auth
        from elasticsearch import Elasticsearch
        from elasticsearch import RequestsHttpConnection
        session = boto3.Session()
        credentials = session.get_credentials()
        aws_auth = AWS4Auth(
            credentials.access_key, credentials.secret_key, 'us-east-1', 'es', session_token=credentials.token)
        self.__elasticsearch_client = Elasticsearch(
            hosts=[{'host': self.__host, 'port': self.__port}],
            http_auth=aws_auth, use_ssl=self.__use_ssl, verify_certs=self.__verify_certs,
            connection_class=RequestsHttpConnection, timeout=30, max_retries=10, retry_on_timeout=True)

    def setup(self, host: str, port: int, use_ssl: bool, verify_certs: bool):
        self.__host: str = host
        self.__port: int = port
        self.__use_ssl: bool = use_ssl
        self.__verify_certs: bool = verify_certs

    @staticmethod
    def get_instance():
        if ElasticsearchClientInstanceGenerator.__instance is None:
            ElasticsearchClientInstanceGenerator()
        return ElasticsearchClientInstanceGenerator.__instance

Then I created this decorator

def try_until_succeed(func):

    def catch_authorization_exception(*args, **kwargs):

        for i in range(0,10):
            try:
                data = func(*args, **kwargs)
                return data
            except AuthorizationException as ae:
               ElasticsearchClientInstanceGenerator.get_instance().generate_elasticsearch_client()
        raise Execption('Elasticsearch general exception')
    return catch_authorization_exception

I'm using the decorator on each methods that require a connection with Elasticcearch. Any time I got the AuthorizationException error, client credentials are refreshed and, after that, I can connect to Elasticsearch.

AuthorizationException: The security token included in the request is expired

1 Answers1