Refreshing boto3 session when paginating though cloudtrail

Question

I'm writing a script in python using boto3 to report on the api calls made over the past few months. I have the script pretty much done but we have a max session length of 1 hour and this will always take longer than that and so the session expires and the script dies.

I have tried to refresh the session periodically to stop it from expiring but I cant't seem to make it work. I'm really hoping that someone has done this before and can tell me what I'm doing wrong?

Below is a cut down version of the code.

import boto3
import datetime
import time
from botocore.exceptions import ClientError

session_start_time = datetime.datetime.now()
start_date = datetime.datetime.now()
start_date -= datetime.timedelta(days=1)
end_date = datetime.datetime.now()
role='arn:aws:iam::1234:role/role'

def role_arn_to_session(**args):
  client = boto3.client('sts')
  response = client.assume_role(**args)

  return boto3.Session(
      aws_access_key_id=response['Credentials']['AccessKeyId'],
      aws_secret_access_key=response['Credentials']['SecretAccessKey'],
      aws_session_token=response['Credentials']['SessionToken'])


session = role_arn_to_session(RoleArn=role,RoleSessionName='session')

cloudtrail = session.client('cloudtrail',region_name='us-east-1')

paginator = cloudtrail.get_paginator("lookup_events")

StartingToken = None
page_iterator = paginator.paginate(
  PaginationConfig={'PageSize':1000, 'StartingToken':StartingToken },
  StartTime=start_date,
  EndTime=end_date)

for page in page_iterator:
  for ct in page['Events']:
    print(ct)
  try:
    token_file = open("token","w")
    token_file.write(page["NextToken"])
    StartingToken = page["NextToken"]

  except KeyError:
    break

  if (datetime.datetime.now() - session_start_time).seconds/60 > 10:
    page_iterator = None
    paginator = None
    cloudtrail = None
    session = None

    session = role_arn_to_session(RoleArn=role,RoleSessionName='session')

    cloudtrail = session.client('cloudtrail',region_name='us-east-1')
    paginator = cloudtrail.get_paginator("lookup_events")
    page_iterator = paginator.paginate(
      PaginationConfig={'PageSize':1000, 'StartingToken':StartingToken },
      StartTime=start_date,
      EndTime=end_date)

    session_start_time = datetime.datetime.now()

I'd appreciate any help with this.

Thanks in advance

*"I can't seem to make it work"* needs to be accompanied by a thorough explanation of how and in what way it fails. — Michael - sqlbot, Oct 20 '19 at 23:42
Sorry, yes that's not very well explained. The problem is that no matter what i try the session always expires and the process ends with a session expired message — Steve, Oct 21 '19 at 06:45
even though the code falls into the if statement that refreshes the session, paginator, etc it's like the page_iterator loop is still using the old session. Therefore this always errors once the initial session has expired. — Steve, Oct 21 '19 at 11:10

Vikyol · Answer 1 · 2019-10-21T12:06:43.743

Your solution does not work because you are just shadowing page_iterator variable, so the changes you make to the iterator does not take effect.

You can increase the session length if you are running your script using long-term credentials.

By default, the temporary security credentials created by AssumeRole last for one hour. However, you can use the optional DurationSeconds parameter to specify the duration of your session.

Otherwise, you need to revise the application logic a bit. You can try using a shorter time frame when fetching the trails, e.g. instead of using 1 day, try using 6 hours and slide the time frame accordingly until you fetch all trails you want. This is a better approach in my opinion.

Thanks, however we have a policy that the maximum session length is 1 hour. If there's actaully no way to refresh the session then i'll have just to only iterate through the trails one day (or maybe one week) at a time. — Steve, Oct 21 '19 at 11:24

Refreshing boto3 session when paginating though cloudtrail

1 Answers1