2

I would like to run some code whenever my AWS EC2 instance starts. The code should pull down data from Amazon S3, do some work on the data, and then copy the data back to S3. I created a makefile to do this, which works fine if I call it while I'm logged into the instance. I then placed a script in /ect/rc.local (this scripts runs every time the instance starts) that will call the makefile. This scripts successfully runs on instance startup. The problem I'm having is that when the makefile is called from the startup script it does not pull data from or copy data to s3. I read here that setting your access keys solves this problem with a Windows server, but this does not work for me. It looks like the code just stops when it tries to call any aws commands because in the log file the output is always the first line of code from the makefile. Below is what my log file says:

aws s3 sync s3:<s3 bucket to get data from> <location to save data to>

Here is the relelvant code from my makefile:

### Download all data
get_data:
        aws s3 sync s3:<s3 bucket to get data from> <location to save data to>

### Copy data back to s3
copy_data_to_s3:
        aws s3 sync <location of data to copy to s3> s3:<s3 bucket data is copied to>

Here is my script in /etc/rc.local:

#!/bin/bash
#
# rc.local
#
make -f <location of makefile>/Makefile > <location to save log file>/log.txt
exit 0

Any help would be appreciated.

Community
  • 1
  • 1
Gabriel
  • 624
  • 1
  • 7
  • 20

1 Answers1

3

When you configure the AWS command line, it stores the credentials and region in ~/.aws/.... But when you execute your command on startup, from rc.local, it's not running as you.

The problem you're seeing is the AWS CLI failing to find any credentials.

So, you have a couple of options:

Option 1: (Preferred)

Don't configure locally-stored AWS credentials. Instead, launch your EC2 instance using an IAM role. When you do this, no credentials are required to be store on your instance. The AWS CLI will simply "find" the credentials in the IAM Role.

Option 2: (May work)

From rc.local, run your scripts under your account. This way, the stored credentials may be found.

The more secure way to do what you want to do is using Option 1.

Matt Houser
  • 33,983
  • 6
  • 70
  • 88
  • Thanks Matt, I'm going to try this now and let you know how it works. – Gabriel Aug 29 '16 at 21:47
  • Well, relative security depends on a few things. The instance cannot be removed from the instance role, so any users who can log on to the instance will have the same AWS resource privileges in the instance role. If the credentials are instead written to a secure place, not all users would be able to leverage the AWS privileges. Option 1 is still better in most situations, but for some use cases, particularly if multiple people with lesser degrees of trust will have shell access on the host, option 2 may be preferable. – Karen B Aug 29 '16 at 22:49
  • I attempted option 1. I created a role and attached AmazonS3FullAccess policy. Additionally, I went to the Policy Simulator and ran getObject and putObject and both showed they were allowed. I created a new instance and attached the role. However, it's still not working. The log file still has the same output and my data is not saved to the instance. I will try option 2 now. – Gabriel Aug 30 '16 at 16:52
  • Option 2 worked. Although I am curious why option 1 didn't. Thank you. – Gabriel Aug 30 '16 at 17:27