failing to decrypt blob passwords only once in a while using amazon kms

Question

import os, sys
AWS_DIRECTORY = '/home/jenkins/.aws'
certificates_folder = 'my_folder'

SUCCESS = 'success'

class AmazonKMS(object):


def __init__(self):
    # making sure boto3 has the certificates and region files
    result = os.system('mkdir -p ' + AWS_DIRECTORY)
    self._check_os_result(result)
    result = os.system('cp ' + certificates_folder + 'kms_config ' + AWS_DIRECTORY + '/config')
    self._check_os_result(result)
    result = os.system('cp ' + certificates_folder + 'kms_credentials ' + AWS_DIRECTORY + '/credentials')
    self._check_os_result(result)


    # boto3 is the amazon client package
    import boto3
    self.kms_client = boto3.client('kms', region_name='us-east-1')
    self.global_key_alias = 'alias/global'
    self.global_key_id = None

def _check_os_result(self, result):
    if result != 0 and raise_on_copy_error:
        raise FAILED_COPY


def decrypt_text(self, encrypted_text):
    response = self.kms_client.decrypt(
        CiphertextBlob = encrypted_text
    )

    return response['Plaintext']

when using it amazon_kms = AmazonKMS() amazon_kms.decrypt_text(blob_password)

getting

E   ClientError: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

stacktrace is

../keys_management/amazon_kms.py:77: in decrypt_text
    CiphertextBlob = encrypted_text
/home/jenkins/.virtualenvs/global_tests/local/lib/python2.7/site-packages/botocore/client.py:253: in _api_call
    return self._make_api_call(operation_name, kwargs)
/home/jenkins/.virtualenvs/global_tests/local/lib/python2.7/site-packages/botocore/client.py:557: in _make_api_call
    raise error_class(parsed_response, operation_name)

This happens in a script that runs once an hour.

it's only failing 2 -3 times a day.

after a retry it succeed.

Tried to upgraded from boto3 1.2.3 to 1.4.4

what is the possible cause for this behavior ?

score 1 · Answer 1 · answered May 06 '20 at 09:22

My guess is that the issue is not in anything you described here. Most likely the login-tokes time out or something along those lines. To investigate this further a closer look on the way the login works here is probably helpful. How does this code run? Is it running inside AWS like on Lambda or EC2? Do you run it from your own server (looks like it runs on jenkins)? How is the login access established? What are those kms_credentials used for and how do they look like? Do you do something like assumeing a role (which would probably work through access tokens which after some time will no longer work)?

failing to decrypt blob passwords only once in a while using amazon kms

1 Answers1