packer provisioning by ansible fails in aws codebuild

Question

My Codebuild project that it creates AMI by packer by ansible provisioner.

This packer settings success in my local environment and Amazon linux2 ec2 environment. However, when I use AWS Codebuild with aws/codebuild/amazonlinux2-x86_64-standard:1.0 image and it fails.

I already tried this settings remote_tmp = /tmp or remote_tmp = /tmp/.ansible-${USER}/tmp but did not work.

Authentication or permission failure, did not have permissions on the remote directory

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.7
  pre_build:
    commands:
      - python --version
      - pip --version
      - curl -qL -o packer.zip https://releases.hashicorp.com/packer/1.4.3/packer_1.4.3_linux_amd64.zip && unzip packer.zip
      - ./packer version
      - pip install --user ansible==2.8.5
      - ansible --version
      - echo 'Validate packer json'
      - ./packer validate packer.json
  build:
    commands:
      - ./packer build -color=false packer.json | tee build.log

{
    "builders": [{
        "type": "amazon-ebs",
        "region": "ap-northeast-1",
        "ami_regions": "ap-northeast-1",
        "source_ami": "ami-0ff21806645c5e492",
        "instance_type": "t2.micro",
        "ssh_username": "ec2-user",
        "ami_name": "packer-quick-start {{timestamp}}",
        "ami_description": "created by packer at {{timestamp}}",
        "ebs_optimized": false,
        "tags": {
            "OS_Version": "Amazon Linux AMI 2018.03",
            "timestamp": "{{timestamp}}",
            "isotime": "{{isotime \"2006-01-02 03:04:05\"}}"
        },
        "disable_stop_instance": false
    }],
    "provisioners": [
        {
            "type" : "ansible",
            "extra_arguments": [
                "-vvv"
            ],
            "playbook_file" : "ansible/main.yaml"
        }
    ]
}

==> amazon-ebs: Prevalidating AMI Name: packer-quick-start 1569943272 
    amazon-ebs: Found Image ID: ami-0ff21806645c5e492 
==> amazon-ebs: Creating temporary keypair: packer_5d936ee8-541f-5c9a-6955-9672526afc1a 
==> amazon-ebs: Creating temporary security group for this instance: packer_5d936ef1-6546-d9d0-60ff-2dc4c011036f 
==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups... 
==> amazon-ebs: Launching a source AWS instance... 
==> amazon-ebs: Adding tags to source instance 
    amazon-ebs: Adding tag: "Name": "Packer Builder" 
    amazon-ebs: Instance ID: i-04b00db56a8b3b6d0 
==> amazon-ebs: Waiting for instance (i-04b00db56a8b3b6d0) to become ready... 
==> amazon-ebs: Using ssh communicator to connect: 3.112.61.8 
==> amazon-ebs: Waiting for SSH to become available... 
==> amazon-ebs: Connected to SSH! 
==> amazon-ebs: Provisioning with Ansible... 
==> amazon-ebs: Executing Ansible: ansible-playbook --extra-vars packer_build_name=amazon-ebs packer_builder_type=amazon-ebs -o IdentitiesOnly=yes -i /tmp/packer-provisioner-ansible244097143 /codebuild/output/src965785042/src/github.com/repoUsername/reponame/ansible/main.yaml -e ansible_ssh_private_key_file=/tmp/ansible-key242793848 -vvv 
    amazon-ebs: ansible-playbook 2.8.5 
    amazon-ebs:   config file = /codebuild/output/src965785042/src/github.com/repoUsername/reponame/ansible.cfg 
    amazon-ebs:   configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] 
    amazon-ebs:   ansible python module location = /root/.local/lib/python3.7/site-packages/ansible 
    amazon-ebs:   executable location = /root/.local/bin/ansible-playbook 
    amazon-ebs:   python version = 3.7.4 (default, Sep 20 2019, 22:55:10) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] 
    amazon-ebs: Using /codebuild/output/src965785042/src/github.com/repoUsername/reponame/ansible.cfg as config file 
    amazon-ebs: host_list declined parsing /tmp/packer-provisioner-ansible244097143 as it did not pass it's verify_file() method 
    amazon-ebs: script declined parsing /tmp/packer-provisioner-ansible244097143 as it did not pass it's verify_file() method 
    amazon-ebs: auto declined parsing /tmp/packer-provisioner-ansible244097143 as it did not pass it's verify_file() method 
    amazon-ebs: Parsed /tmp/packer-provisioner-ansible244097143 inventory source with ini plugin 
    amazon-ebs: 
    amazon-ebs: PLAYBOOK: main.yaml ************************************************************ 
    amazon-ebs: 1 plays in /codebuild/output/src965785042/src/github.com/repoUsername/reponame/ansible/main.yaml 
    amazon-ebs: 
    amazon-ebs: PLAY [all] ********************************************************************* 
    amazon-ebs: META: ran handlers 
    amazon-ebs: 
    amazon-ebs: TASK [be sure httpd is installed] ********************************************** 
    amazon-ebs: task path: /codebuild/output/src965785042/src/github.com/repoUsername/reponame/ansible/main.yaml:6 
    amazon-ebs: <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root 
    amazon-ebs: <127.0.0.1> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=35595 -o 'IdentityFile="/tmp/ansible-key242793848"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/02aaab1733 127.0.0.1 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"'' 
    amazon-ebs: <127.0.0.1> (0, b'/root\n', b"Warning: Permanently added '[127.0.0.1]:35595' (RSA) to the list of known hosts.\r\n") 
    amazon-ebs: <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root 
    amazon-ebs: <127.0.0.1> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=35595 -o 'IdentityFile="/tmp/ansible-key242793848"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/02aaab1733 127.0.0.1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1569943320.4544108-49329379039882 `" && echo ansible-tmp-1569943320.4544108-49329379039882="` echo /root/.ansible/tmp/ansible-tmp-1569943320.4544108-49329379039882 `" ) && sleep 0'"'"'' 
    amazon-ebs: <127.0.0.1> (1, b'', b'mkdir: cannot create directory \xe2\x80\x98/root\xe2\x80\x99: Permission denied\n') 
    amazon-ebs: <127.0.0.1> Failed to connect to the host via ssh: mkdir: cannot create directory ‘/root’: Permission denied 
    amazon-ebs: fatal: [default]: UNREACHABLE! => { 
    amazon-ebs:     "changed": false, 
    amazon-ebs:     "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /root/.ansible/tmp/ansible-tmp-1569943320.4544108-49329379039882 `\" && echo ansible-tmp-1569943320.4544108-49329379039882=\"` echo /root/.ansible/tmp/ansible-tmp-1569943320.4544108-49329379039882 `\" ), exited with result 1", 
    amazon-ebs:     "unreachable": true 
    amazon-ebs: } 
    amazon-ebs: 
    amazon-ebs: PLAY RECAP ********************************************************************* 
    amazon-ebs: default                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0 
    amazon-ebs: 
==> amazon-ebs: Terminating the source AWS instance... 
==> amazon-ebs: Cleaning up any extra volumes... 
==> amazon-ebs: No volumes to clean up, skipping 
==> amazon-ebs: Deleting temporary security group... 
==> amazon-ebs: Deleting temporary keypair...

I know it fails because it tried to mkdir /root and Permission denied. But don't know why it tried to mkdir /root. How can I change this behavior?

It is likely due to the SSH agent necessary for Ansible communication. This is also likely an Ansible issue and not Packer or AWS. — Matthew Schuchard, Oct 01 '19 at 16:20
Thank you. So maybe I should try debug and see differences between my laptop and codebuild — db099u, Oct 02 '19 at 00:04
Looks more like ansible playbook issue. Can you share "ansible/main.yaml" — shariqmaws, Oct 02 '19 at 13:29

score 8 · Answer 1 · answered Oct 02 '19 at 14:04

8

I solved and it was super simple cause. Because AWS Codebuild builds by the root user, ansible makes a connection by the root user. I just wrote like this and solved it.

    "provisioners": [
        {
            "type" : "ansible",
            "user": "ec2-user",
            "playbook_file" : "ansible/main.yaml"
        }
    ]

My ansible file is simple for testing.

---
- hosts: all
  become: yes
  gather_facts: no
  tasks:
    - name: be sure httpd is installed
      yum: name=httpd state=installed
    - name: be sure httpd is running and enabled
      service: name=httpd state=started enabled=yes

answered Oct 02 '19 at 14:04

db099u

505
1
5
12

I'm not sure how you got this far because as root use I find I just have problems with Ansible not being on the path (I'm also using codebuild and packer). "executable file not found in $PATH". – openCivilisation Jan 10 '22 at 07:36
Thanks! For some reason, this worked on macOS, but not on my Jenkins' Kubernetes build's pod. I am super confused though why it even "worked on my machine" in the first place since my username is not `ubuntu`. I think for some reason it worked locally because I specified `ssh_username` in my source.pkr.hcl file. :/ – Robert J Mar 29 '22 at 07:42

packer provisioning by ansible fails in aws codebuild

1 Answers1

Linked