I am currently working with a log collection product and want to be able to pull in my CloudTrail logs from AWS. I started using the boto3 client in order to lookup the events in CloudTrail. I got the script to work right when I am running it directly from the commandline, but as soon as I tried to put it in cron to pull the logs automatically over time, it stopped collecting the logs!
Here's a sample of the basics of what's in the script to pull the logs:
#!/usr/bin/python
import boto3
import datetime
import json
import time
import sys
import os
def initialize_log():
try:
log = open('/var/log/aws-cloudtrail.log', 'ab')
except IOError as e:
print " [!] ERROR: Cannot open /var/log/aws-cloudtrail.log (%s)" % (e.strerror)
sys.exit(1)
return log
def date_handler(obj):
return obj.isoformat() if hasattr(obj, 'isoformat') else obj
def read_logs(log):
print "[+] START: Connecting to CloudTrail Logs"
cloudTrail = boto3.client('cloudtrail')
starttime = ""
endtime = ""
if os.path.isfile('/var/log/aws-cloudtrail.bookmark'):
try:
with open('/var/log/aws-cloudtrail.bookmark', 'r') as myfile:
strdate=myfile.read().replace('\n', '')
starttime = datetime.datetime.strptime( strdate, "%Y-%m-%dT%H:%M:%S.%f" )
print " [-] INFO: Found bookmark! Querying with a start time of " + str(starttime)
except IOError as e:
print " [!] ERROR: Cannot open /var/log/aws-cloudtrail.log (%s)" % (e.strerror)
else:
starttime = datetime.datetime.now() - datetime.timedelta(minutes=15)
print " [-] INFO: Cannot find bookmark...Querying with start time of" + str(starttime)
endtime = datetime.datetime.now()
print " [-] INFO: Querying for CloudTrail Logs"
response = cloudTrail.lookup_events(StartTime=starttime, EndTime=endtime, MaxResults=50)
for event in response['Events']:
log.write(json.dumps(event, default=date_handler))
log.write("\n")
print json.dumps(event, default=date_handler)
print "------------------------------------------------------------"
if 'NextToken' in response.keys():
while 'NextToken' in response.keys():
time.sleep(1)
response = cloudTrail.lookup_events(StartTime=starttime, EndTime=endtime, MaxResults=50, NextToken=str(response['NextToken']))
for event in response['Events']:
log.write(json.dumps(event, default=date_handler))
log.write("\n")
print json.dumps(event, default=date_handler)
print "------------------------------------------------------------"
# log.write("\n TESTING 1,2,3 \n")
log.close()
try:
bookmark_file = open('/var/log/aws-cloudtrail.bookmark','w')
bookmark_file.write(str(endtime.isoformat()))
bookmark_file.close()
except IOError as e:
print " [!] ERROR: Cannot set bookmark for last pull time in /var/log/aws-cloudtrail.bookmark (%s)" % (e.strerror)
sys.exit(1)
return True
log = initialize_log()
success = read_logs(log)
if success:
print "[+] DONE: All results printed"
else:
print "[+] ERROR: CloudTrail results were not able to be pulled"
I looked into it more and did some testing to confirm that permissions were right on the destination files and that the script could write to them when run from root's crontab, but I still wasn't getting the logs returned from the boto cloudtrail client unless I ran it manually.
I also checked to make sure that the default region was getting read correctly from /root/.aws/config and it looks like it is, because if I move it I see the cron email show a stack trace instead of the success messages I have built in.
I am hoping someone has already run into this and it's a quick simple answer!
EDIT: The permissions to the cloudtrail logs is allowed via the instance's IAM Role, and yes, the task is scheduled under root's crontab.
Here's the email output:
From root@system Mon Mar 28 23:00:02 2016
X-Original-To: root
From: root@system (Cron Daemon)
To: root@system
Subject: Cron <root@system> /usr/bin/python /root/scripts/get-cloudtrail.py
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Date: Mon, 28 Mar 2016 19:00:02 -0400 (EDT)
[+] START: Connecting to CloudTrail Logs
[-] INFO: Found bookmark! Querying with a start time of 2016-03-28 22:55:01.395001
[-] INFO: Querying for CloudTrail Logs
[+] DONE: All results printed
