Ansible to access Github: become causes Permission denied (publickey)

Question

Objective

Would like to understand the mechanism why using become to be the ansible user causes "Permission denied (publickey)".

User ansible to run ansible-playbook to checkout a Github repository. SSH keys have been copied with ssh-copy-id with the user.

Without become, the playbook runs.

[ansible@ip-172-31-39-108 playbooks]$ whoami
ansible

[ansible@ip-172-31-39-108 playbooks]$ ansible-playbook git.yml
PLAY [Git example] *************************************************************
TASK [setup] *******************************************************************
ok: [ub01]
TASK [check out the repository on the host] ************************************
changed: [ub01]
PLAY RECAP *********************************************************************
ub01                       : ok=2    changed=1    unreachable=0    failed=0

However, using "become: yes" causes the error.

[ansible@ip-172-31-39-108 playbooks]$ ansible-playbook git.yml
PLAY [Git example] *************************************************************
TASK [setup] *******************************************************************
ok: [ub01]
TASK [check out the repository on the host] ************************************
fatal: [ub01]: FAILED! => {"changed": false, "cmd": "/usr/bin/git clone --origin origin '' /home/ansible/project/mezzanine-example", "failed": true, "msg": "Cloning into '/home/ansible/project/mezzanine-example'...\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.", "rc": 128, "stderr": "Cloning into '/home/ansible/project/mezzanine-example'...\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n", "stdout": "", "stdout_lines": []}
        to retry, use: --limit @/home/ansible/playbooks/git.retry

PLAY RECAP *********************************************************************
ub01                       : ok=1    changed=0    unreachable=0    failed=1

ssh-agent is running and the private key has been added.

[ansible@ip-172-31-39-108 playbooks]$ eval $(ssh-agent -s)
Agent pid 1513

[ansible@ip-172-31-39-108 playbooks]$ ssh-add ~/.ssh/id_rsa
Identity added: /home/ansible/.ssh/id_rsa (/home/ansible/.ssh/id_rsa)

Question

Please provide explanations why this happen or point to the resource to look into.

Also I only started ssh-agent in the server that runs ansible-playbook, but not on the target server. How the Github SSH authentication happens on the target server?

Playbook

- name: Git example
  hosts: webservers
  become: no    # <----- Changing to yes cause the issue
  become_user: ansible
  become_method: sudo
  vars:
    repo_url: git@github.com:lorin/mezzanine-example.git
    proj_dirname: /home/ansible/project
    proj_name: mezzanine-example
    proj_path: "{{ proj_dirname }}/{{ proj_name }}"
  tasks:
    - name: check out the repository on the host
      git: repo={{ repo_url }} dest={{ proj_path }} accept_hostkey=yes

Configurations

hosts

[webservers]
ub01
#rh01

ansible.cfg

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes

Environment

Ansible playbook is run on RedHat.

NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"

The target host is Ubuntu.

DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

Related Issues

Console -vvvvv output

TASK [check out the repository on the host] ************************************
task path: /home/ansible/playbooks/git.yml:12
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/source_control/git.py
<ub01> ESTABLISH SSH CONNECTION FOR USER: None
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set ssh_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r ub01 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1485919043.94-240537002849590 `" && echo ansible-tmp-1485919043.94-240537002849590="` echo /tmp/ansible-tmp-1485919043.94-240537002849590 `" ) && sleep 0'"'"''
<ub01> PUT /tmp/tmpAjaOMc TO /tmp/ansible-tmp-1485919043.94-240537002849590/git.py
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set sftp_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC sftp -b - -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r '[ub01]'
<ub01> ESTABLISH SSH CONNECTION FOR USER: None
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set ssh_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r ub01 '/bin/sh -c '"'"'setfacl -m u:ansible:r-x /tmp/ansible-tmp-1485919043.94-240537002849590/ /tmp/ansible-tmp-1485919043.94-240537002849590/git.py && sleep 0'"'"''
<ub01> ESTABLISH SSH CONNECTION FOR USER: None
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set ssh_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r -tt ub01 '/bin/sh -c '"'"'sudo -H -S -n -u ansible /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-cxuzmrsbxdvydelfnjrsmgvocgkeptxd; /usr/bin/python /tmp/ansible-tmp-1485919043.94-240537002849590/git.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<ub01> ESTABLISH SSH CONNECTION FOR USER: None
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set ssh_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r ub01 '/bin/sh -c '"'"'rm -f -r /tmp/ansible-tmp-1485919043.94-240537002849590/ > /dev/null 2>&1 && sleep 0'"'"''
fatal: [ub01]: FAILED! => {
    "changed": false,
    "cmd": "/usr/bin/git clone --origin origin '' /home/ansible/project/mezzanine-example",
    "failed": true,
    "invocation": {
        "module_args": {
            "accept_hostkey": true,
            "bare": false,
            "clone": true,
            "depth": null,
            "dest": "/home/ansible/project/mezzanine-example",
            "executable": null,
            "force": false,
            "key_file": null,
            "recursive": true,
            "reference": null,
            "refspec": null,
            "remote": "origin",
            "repo": "git@github.com:lorin/mezzanine-example.git",
            "ssh_opts": null,
            "track_submodules": false,
            "umask": null,
            "update": true,
            "verify_commit": false,
            "version": "HEAD"
        },
        "module_name": "git"
    },
    "msg": "Cloning into '/home/ansible/project/mezzanine-example'...\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.",
    "rc": 128,
    "stderr": "Cloning into '/home/ansible/project/mezzanine-example'...\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n",
    "stdout": "",
    "stdout_lines": []
}
        to retry, use: --limit @/home/ansible/playbooks/git.retry

PLAY RECAP *********************************************************************
ub01                       : ok=1    changed=0    unreachable=0    failed=1

Update

Thanks to the answer from @Jakuje and other articles, understood that the SSH agent listens on a UNIX socket file. The filename is stored in SSH_AUTH_SOCK environment variable. However, the SUDO eliminates the environment variables, hence the sudo-ed SSH client does not know how to talk to SSH agent. Therefore, cannot pass the SSh authentication.

The solution is in the articled provided by @Jakuje.

output of `ansible --version` please, also output of `ansible -m shell -a "ssh-add -L" -kKuansible ub01` — Jacob Evans, Feb 01 '17 at 04:05
and also, is your ssh-key for auth and ssh-key for git the same. — Jacob Evans, Feb 01 '17 at 04:06

score 1 · Accepted Answer · edited May 23 '17 at 11:33

When you forward ssh-agent socket, it is created with the privileges of the user who is connecting. become later makes ansible to change the user to some different user (ansible) using sudo, which causes:

The target user does not have access to the forwarded socket
The environment variable "holding the connection" to the agent socket is not preserved during sudo

If the target user is root, the first should not be a problem. The second problem can be resolved by modifying /etc/sudoers on the server to have a line

 Defaults    env_keep += "SSH_AUTH_SOCK"

It is further explained in the post on SO.

To overcome the second problem will be more complicated, because you don't want to have your forwarded socket available to all users. Did you consider connecting directly as a user ansible?