4

Has anyone noticed some unpredictable failures on AWS Secret Manager when trying to retrieve secret values? I'm using my own encryption key, and I've found that frequently, I am getting a "Failed to fetch a list of secrets" error on the AWS console after encrypting a secret. This seems to happen if I change the encryption key after an initial encryption, but it has happened without that as well.

I also think I've seen a case where the encryption key changed from a custom key to default without any action from me.

I've also seen an issue where two stacks set up nearly identically have an inconsistency where one can read an encryption key when calling Secrets Manager but one cannot. It looks like an IAM issue, but I haven't found any difference between the two stacks and their IAM settings. I only mention this in case it gives some clue to the issue above.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
SteveM
  • 66
  • 1
  • 3

2 Answers2

1

I am seeing the same thing as well after I changed the encryption key. I don't understand why this is happening. I will open a ticket with AWS and report back.

OK after talking to AWS Support the issue seems to be a bug. If you disabled (or marked it for deletion) your old encryption key than you will experience this issue.

To fix this you will need to cancel the deletion of your old encryption key, AND change its status to "Enabled". After this you will be able to retrieve your secrets using your new encryption key.

Unfortunately, this is the current workaround until AWS has a permanent solution.

Hope this helps.

Wolverine313
  • 301
  • 2
  • 8
  • 2
    If you change the encryption key, it does not automatically re-encrypt all the secret versions with the new key, they remain encrypted with the old key. So if you delete the old key, you lose access to all the secret versions encrypted with that old key. When you change the key, you will need to re-encrypt the secret (read and re-write) to get it encrypted with the new key. You can then go and delete the old key. – JoeB Mar 20 '19 at 17:17
0

There is not enough data here to provide a reliable answer. However, since you mention stacks and IAM users, I suspect you may be seeing a propagation issue.

Most AWS services, and IAM in particular, are eventually consistent. If you create a user or add permissions to a user, it can take some time for those user permissions to propagate. Usually this happens in seconds, but can sometimes take minutes. Since these are distributed systems, you could hit a node that has your recent permission updates and then hit a node that does not. A good clue is if this all clears up five or ten minutes after you have created everything.

JoeB
  • 1,503
  • 7
  • 9