Avoid pre-signed URL expiry when IAM role key rotates

Question

In Airflow I have 2 tasks defined that run every day:

the first one creates a zip file and saves it in AWS under s3://{bucket-name}/foo/bar/{date}/archive.zip
the second one pre-signs that url (should expire in 7 days) and sends it to Slack.

Because Qubole uses an IAM role the generated url will expire when the keys are rotated (less than 24 hours as far as I can tell).

I'm trying to find a solution for this. My current idea is moving the second task into an AWS lambda and using IAM user credentials to avoid the expiry issue.

Is there any other approach I could take without over complicating it?

Wanted to follow up and see if you had come up with any solution for this? We want to move properly restricted rotating IAM roles but for now want to keep backwards compatibility with some permanently-cached presigned URLs that we currently have. We are considering Lambda as well. — scorgn, Oct 28 '22 at 18:34

score 1 · Answer 1 · answered May 12 '20 at 13:20

1

You will need to use specific IAM credentials. Best practice would be be setting the permissions on the IAM user to only do what is required.

answered May 12 '20 at 13:20

Chris Williams

32,215
4
30
68

Do you mean creating a specific IAM user (with the required actions) for the AWS lambda? – Maria Livia May 12 '20 at 13:26
1

The Lambda will need to generate the self signed URL using the IAM key. (Lambda should still have its own execution role). Store the IAM key and secret as environment variables or store in SSM or Secrets Manager – Chris Williams May 12 '20 at 13:27
1

I see, I'll give that a go. Thanks! – Maria Livia May 12 '20 at 13:30

Avoid pre-signed URL expiry when IAM role key rotates

1 Answers1

Linked