How to verify the Secrets Manager credential rotation is successful

Question

I enabled the rotation and checked lambda funcion logs, there is no error, everything looks fine. But I retrieved the secret in Secrets Manager console, it's still the old password. I don't know what's going on.

In AWS CLI:

aws secretsmanager list-secret-version-ids --secret-id xxx

Output:
{
    "Versions": [
        {
            "VersionId": "4********2f",
            "VersionStages": [
                "AWSPREVIOUS"
            ],
            "LastAccessedDate": 1580428800.0,
            "CreatedDate": 1580484370.988
        },
        {
            "VersionId": "9********69",
            "VersionStages": [
                "AWSPENDING"
            ],
            "LastAccessedDate": 1580428800.0,
            "CreatedDate": 1580484483.303
        },
        {
            "VersionId": "b5*********c5",
            "VersionStages": [
                "AWSCURRENT"
            ],
            "LastAccessedDate": 1580428800.0,
            "CreatedDate": 1580484482.179
        }
    ],
    "ARN": "arn:aws:secretsmanager:us-east-1:xxxxxxxxxxxxxx",
    "Name": "xxxxxxxxxxxxxx"
}

I see a AWSPENDINGabove, what does it mean? I notice the version ID changed, does it mean the rotation is successful? Many thanks.

Additional info: In cloudwatch logs:

[INFO]  2020-02-01T19:30:47.741Z    3d6XXX81    Found credentials in environment variables.
[INFO]  2020-02-01T19:30:48.416Z    3dXXX81 createSecret: Successfully retrieved secret for arn:XXX
END RequestId: 3dXXX81
REPORT RequestId: 3dXXX81
START RequestId: adXXX52 Version: $LATEST
END RequestId: adXXX52
REPORT RequestId: adXXX52
2020-02-01T19:30:51.485Z adXXX52 Task timed out after 3.00 seconds

I started a new rotation manually, and it gave me a new error:

[ERROR] ValueError: Unable to log into database with previous, current, or pending secret of secret xxxxxx

When I tried to access to MySQL db I couldn't login by using the original password, does this mean the password has been rotated to a random value and it's 'PENDING' now? Not sure what to do now, is there a way to reset the password?

Retriggered another rotation, it gave me unable to log into db and [ERROR] ClientError: An error occurred (AccessDeniedException) when calling the GetSecretValue operation:

I really don't understand why I kept getting new errors, I'm pretty sure I added 'GetSecretValue' in the Lambda exec role, anyone knows what the hell is going on?

Can you check to verify that you see this message in your CW lambda logs ?- https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/blob/master/SecretsManagerRDSMySQLRotationSingleUser/lambda_function.py#L248 — committedandroider, Jan 31 '20 at 19:30
Now the logs is not complaining with errors, the last info in the logs is something like: ```Found credentials in environment variables```. and then ```createSecret: Successfully retrieved secret for arn:xxxx```, seems like it stops after creating the secret, any idea what's going on here? @committedandroider I noticed that you answered almost all of my questions about credential rotation, do you mind sending me an email address so that we can discuss something? I've been working on this task for a week now, still haven't solved it.... — wawawa, Feb 03 '20 at 09:32
@committedandroider I've added the logs outputs in the question, thank you. — wawawa, Feb 03 '20 at 09:38
Hi you are right, I did get the error you mentioned ```setSecret: Unable to log into database with previous, current, or pending secret```, this honestly drives me crazy now, I got various errors in the logs..... @committedandroider — wawawa, Feb 03 '20 at 14:19
I am facing s similar task time out with no clear error like the one mentioned in the question. Where should I look for further troubleshooting? — hyperwiser, Aug 13 '20 at 20:03

score 0 · Answer 1 · edited Jan 29 '23 at 12:54

Yes. Your password is rotated. U can retrieve your credentials from secret manager. After retrieving password from secret manager, log into DB with that new password. If its not working modify the master password in DB, by using this new password taken from secret manager.

Additionally, you can check your secret manager and other activities in AWS CLOUDTRAIL Event history. In Event History, if the secrets are rotated successfully, it shows ROTATION SUCEEDED.

How to verify the Secrets Manager credential rotation is successful

1 Answers1

Linked