2

I am trying to get list of all the available snapshots in AWS account. I have written following code.

def lambda_handler(event, context):
  ec2 = boto3.client('ec2')
  resp_describe_snapshots = ec2.describe_snapshots(OwnerIds=['self'])
  snapshot =  resp_describe_snapshots['Snapshots']

  snapshots = {}
  os.chdir('/tmp')

  for snapshotIdList in resp_describe_snapshots['Snapshots']:
    snapid =  snapshotIdList.get('SnapshotId')
    Des = snapshotIdList.get('Description')
    sttime = snapshotIdList.get('StartTime').strftime("%b %d %Y")

    #print(snapshotIdList.get('SnapshotId')+ ","+ snapshotIdList.get('Description')+ ","+ snapshotIdList.get('StartTime').strftime("%b %d %Y"))    

  for response in ec2.get_paginator('describe_snapshots').paginate(OwnerIds=['self'],Filters=[{'Name': 'tag:Name','Values': ['Ubuntu']}]):
    snapshots.update([(snapshot['SnapshotId'], snapshot) for snapshot in response['Snapshots']])
    print(snapshots)

How can I list all the snapshots ID and it's creation date ?

I also want to create a csv file.

cloud_geek
  • 335
  • 3
  • 13

1 Answers1

2

Here is my example of pagination.

import boto3

boto3 = boto3.session.Session(region_name='ap-northeast-2')
ec2 = boto3.client('ec2')

page_iterator = ec2.get_paginator('describe_snapshots').paginate()

for page in page_iterator:
    for snapshot in page['Snapshots']:
        print(snapshot['SnapshotId'], snapshot['StartTime'])

The result is

snap-5... 2015-MM-dd 18:21:59+00:00
snap-e... 2016-MM-dd 22:15:38+00:00
snap-e... 2016-MM-dd 22:15:46+00:00
snap-e... 2017-MM-dd 12:59:00+00:00
snap-e... 2016-MM-dd 20:39:12+00:00
snap-8... 2017-MM-dd 13:04:34+00:00
snap-7... 2017-MM-dd 17:28:49+00:00
snap-b... 2016-MM-dd 14:52:38+00:00
snap-b... 2016-MM-dd 21:18:23+00:00
snap-d... 2016-MM-dd 07:43:38+00:00
snap-2... 2015-MM-dd 00:07:19+00:00
snap-b... 2017-MM-dd 22:03:26+00:00
snap-0... 2016-MM-dd 00:32:35+00:00
snap-c... 2016-MM-dd 21:57:05+00:00
snap-7... 2016-MM-dd 04:10:45+00:00
...

where the number of results are over 10,000.

To make csv, 

import boto3
import csv

boto3 = boto3.session.Session(region_name='ap-northeast-2')
ec2 = boto3.client('ec2')

with open('snapshots.csv', 'w') as csvfile: 
    fields = ['SnapshotId', 'StartDate'] 
    writer = csv.writer(csvfile)
    writer.writerow(fields) 

    page_iterator = ec2.get_paginator('describe_snapshots').paginate()

    for page in page_iterator:
        for snapshot in page['Snapshots']:
            temp = [snapshot['SnapshotId'], snapshot['StartTime']]
            writer.writerows([temp])

csvfile.close()

To update csv into S3 directly,

import boto3
import csv
import io

boto3 = boto3.session.Session(region_name='ap-northeast-2')
ec2 = boto3.client('ec2')
s3 = boto3.client('s3')

fields = ['SnapshotId', 'StartDate'] 
csvio = io.StringIO()
writer = csv.writer(csvio)
writer.writerow(fields) 

page_iterator = ec2.get_paginator('describe_snapshots').paginate()

for page in page_iterator:
    for snapshot in page['Snapshots']:
        temp = [snapshot['SnapshotId'], snapshot['StartTime']]
        writer.writerows([temp])


s3.put_object(Body=csvio.getvalue(), ContentType='application/vnd.ms-excel', Bucket='bucket', Key='snapshot.csv') 
csvio.close()
Lamanus
  • 12,898
  • 4
  • 21
  • 47
  • Thanks....I will execute it using AWS lambda..... Have you pushed the result into CSV file ? I have following code.....Is this the proper way to push results in csv ? with open('snapshots.csv', 'w', newline="") as csvfile: fields = ['SnapshotId', 'StartDate'] writer = csv.writer(csvfile) writer.writerow(fields) list1 = [snapshot['SnapshotId'], snapshot['StartTime']] writer.writerows(list1) – cloud_geek Aug 29 '19 at 08:41
  • Thanks..... I created the same code..... Have you pushed this file to s3 bucket ? – cloud_geek Aug 29 '19 at 09:13
  • 1
    add `s3 = boto3.cliend('s3')` below ec2, and add a line to the final `s3.upload_file('snapshots.csv', 'bucket', 'object_name')` – Lamanus Aug 29 '19 at 09:18
  • 1
    Check [this](https://stackoverflow.com/questions/4993439/how-can-i-access-s3-files-in-python-using-urls) for other solutions. you can even use s3fs package to read and write the file directly from s3. – Lamanus Aug 29 '19 at 09:19
  • Actually I am trying it from lambda function....I used s3.upload_file(file_name, bucket_name, file_name) but it is uploading blank file..... DO we need to give any content type ? – cloud_geek Aug 29 '19 at 09:22
  • Specify the path. When you save the csv, `/tmp/snapshot.csv` and upload_file from this path. The other folder paths are not permitted to access in Lambda I guess. – Lamanus Aug 29 '19 at 09:35
  • I tried giving /tmp/ path but still it is uploading blank file....I found one code in github where it is showing similar kind of code......but it is not working.....This is the code from io import BytesIO csv_buffer = BytesIO() csv_df.to_csv(csv_buffer, index=False) s3client = boto3.client("s3") response = s3client.put_object( Body=csv_buffer.getvalue(), ContentType='application/vnd.ms-excel', Bucket=bucket, Key=output_file_path, ) – cloud_geek Aug 29 '19 at 09:43
  • 1
    Added new code by using `StringIO()`. Try this. If this is still empty, then check your snapshots filtering condition which may not give correct result. – Lamanus Aug 29 '19 at 10:11
  • Yes...It is appending the list to csv file.........but It is adding every single character to new column... ex. If snapshot is is snap-abcd1234 then 's' , 'n', 'a', 'p', ............. is getting added to new column.... Will check how to put 1st entire list value to single column – cloud_geek Aug 29 '19 at 11:13
  • 1
    have you checked my code? `writer.writerows([temp])` – Lamanus Aug 29 '19 at 11:17
  • 1
    Yes....Now it's working fine for me.......... Thanks a lot for your help ! – cloud_geek Aug 29 '19 at 11:45
  • actually I tried with your code.........now everything is getting written to single column only – cloud_geek Aug 29 '19 at 13:08
  • How can we write this result into excel sheet instead of creating csv file ? – cloud_geek Sep 03 '19 at 05:43