EC2 User Data not working via python boto command

Question

I am trying to launch an instance, have a script run the first time it launches as part of userdata. The following code was used (python boto3 library):

import boto3
ec2 = boto3.resource('ec2')
instance = ec2.create_instances(DryRun=False, ImageId='ami-abcd1234', MinCount=1, MaxCount=1, KeyName='tde', Placement={'AvailabilityZone': 'us-west-2a'}, SecurityGroupIds=['sg-abcd1234'], UserData=user_data, InstanceType='c3.xlarge', SubnetId='subnet-abcd1234')

I have been playing around with the user_data and have had no success. I have been trying to echo some string to a new file in an existing directory. Below is the latest version I attempted.

user_data = '''
    #!/bin/bash
    echo 'test' > /home/ec2/test.txt
    '''

The ami is a CentOS based private AMI. I have tested the commands locally on the server and gotten them to work. But when I put the same command on the userdata (tweaked slightly to match the userdata format), it does not work. Instance launches successfully but the file I specified is not present.

I looked at other examples (https://github.com/coresoftwaregroup/boto-examples/blob/master/32-create-instance-enhanced-with-user-data.py) and even copied their commands.

Your help is appreciated! Thanks :)

Answer 1

For User Data to be recognized as a script, the very first characters MUST be #! (at least for Linux instances).

However, your user_data variable is being defined as:

"\n    #!/bin/bash\n    echo 'test' > /home/ec2/test.txt\n    "

You should define it like this:

user_data = '''#!/bin/bash
echo 'test' > /tmp/hello'''

Which produces:

"#!/bin/bash\necho 'test' > /tmp/hello"

That works correctly.

So, here's the final product:

import boto3

ec2 = boto3.resource('ec2')

user_data = '''#!/bin/bash
echo 'test' > /tmp/hello'''

instance = ec2.create_instances(ImageId='ami-abcd1234', MinCount=1, MaxCount=1, KeyName='my-key', SecurityGroupIds=['sg-abcd1234'], UserData=user_data, InstanceType='t2.nano', SubnetId='subnet-abcd1234')

After logging in:

[ec2-user@ip-172-31-2-151 ~]$ ls /tmp
hello  hsperfdata_root

Answer 2

Maybe your script did not worked because you typed the wrong user path.

The correct user path is /home/ec2-user, and not /home/ec2 as supposed in your script.

Try this as user data:

user_data = '''
    #!/bin/bash
    echo 'test' > /home/ec2-user/test.txt
    '''

Answer 3

I believe you need to specify an instance profile. Have a look at this page: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-api-cli

The steps to do this include:

• Create a role
• Attach the appropriate policy document to that role (note: AmazonEC2FullAccess policy as used in the example might not be it, I just haven't found a better alternative yet)
• Create an instance profile
• Attach the role to the instance profile
• Create instances using the instance profile

Additionally as mentioned in other answers start your user data with

#!/bin/bash

Here is a "simplified" code snippet, I'm using to do the same. This skips over other initialization steps such as picking a machine image, and setting up a security group and such.

iam_client = boto3.client('iam')

# create assume role policy document
assume_role_policy_document = json.dumps({
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Principal": {
            "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
        }
    ]
})

# create ec2 full access role
role_response = iam_client.create_role(
    RoleName='ec2_full_access_role',
    AssumeRolePolicyDocument=assume_role_policy_document,
    Description='AmazonEC2FullAccess',
    MaxSessionDuration=3600,
)

# get policy arm
policy_arn = policies_df[policies_df.PolicyName=='AmazonEC2FullAccess'].Arn.iloc[0]

# attach policy
role_attach_response = iam_client.attach_role_policy(
    RoleName='ec2_full_access_role', 
    PolicyArn=policy_arn
)

# create instance profile
instance_profile_response = iam_client.create_instance_profile(
    InstanceProfileName='ec2_instance_profile',
)

# attach role to instance profile
attach_role_response = iam_client.add_role_to_instance_profile(
    InstanceProfileName='ec2_instance_profile',
    RoleName='ec2_full_access_role'
)

# define user data
user_data = """#!/bin/bash
touch test.txt
"""

# create instance
instance_params = {
    'ImageId': image_id,
    'InstanceType': instance_type,
    'KeyName': key_name,
    'SecurityGroups': (sg_name, ),
    'UserData': user_data,
    'IamInstanceProfile':{'Name': 'ec2_instance_profile'},
    'MinCount': 1, 
    'MaxCount': 1
}
instances = ec2_res.create_instances(**instance_params)

Answer 4

In case anyone is attempting to make user_data work with the amazon.aws.ec2_instance Ansible module, for bootstrapping EKS worker nodes:

user_data: "#!/bin/bash \n
            sudo /etc/eks/bootstrap.sh 
            --apiserver-endpoint {{ CLUSTER_ENDPOINT }}
            --b64-cluster-ca {{ CERTIFICATE_AUTHORITY_DATA }} 
            {{ CLUSTER_NAME }}"

The newline is the important part here.