Setting up SSH jump box for Ansible - can't connect

Question

I'm trying to use Ansible to deploy to an AWS host that does not have a public IP address. I'm able to reach the host manually by ssh onto a jump box, and on that box ssh onto the private machine like my machine-->bastion-->private server

I don't think that I can use Ansible native support because this playbook uses roles that are shared with other plays. These roles rely on particular inventory groups. If I set up group_vars then this will bork the playbooks for deploying this to non-AWS infra.

My ssh config file looks like this:

# Servers in availability zone A
    Host 10.0.0.*
      ProxyCommand ssh -W %h:%p bastion.example.com
      IdentityFile ~/.ssh/key.pem

# Servers in availability zone B
Host 10.0.1.*
  ProxyCommand ssh -W %h:%p bastion.example.com
  IdentityFile ~/.ssh/key.pem

# The bastion host itself
Host bastion.example.com
  User ubuntu
  IdentityFile ~/.ssh/key.pem
  ControlMaster auto
  ControlPath ~/.ssh/ansible-%r@%h:%p
  ControlPersist 5m

Note that the key is the same for the bastion and private servers.

When I try to ssh 10.0.0.175 -F /tmp/aws_bastion_ssh_config -vvv I get the following output:

    (venv) andrew@dell:~/projects/ansible-playbooks$ ssh 10.0.0.175 -F /tmp/aws_bastion_ssh_config -vvv
OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /tmp/aws_bastion_ssh_config
debug1: /tmp/aws_bastion_ssh_config line 6: Applying options for 10.0.0.*
debug1: Executing proxy command: exec ssh -W 10.0.0.175:22 bastion.example.com
debug1: permanently_drop_suid: 1000
debug1: key_load_public: No such file or directory
debug1: identity file /home/andrew/.ssh/key.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/andrew/.ssh/key.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
Permission denied (publickey).
ssh_exchange_identification: Connection closed by remote host

How can I get this to work?

Do I need the key to be present on the jump box? How do I configure that?

Edit: More info:- To clarify that I can connect to the private machine from the jump box when I do ssh bastion.example.com -F /tmp/aws_bastion_ssh_config I reach the bastion server. I've copied the key to that server and when I ssh ubuntu@10.0.0.175 -i ~/.ssh/key.pem I connect to the private machine. Ideally I would prefer not to have the key on the jump box, I've only put it there to make sure the networking is running correctly.

Answer 1

What you do with ProxyCommand is starting two entirely separate ssh commands.

1. that is trying to use the Proxy to authenticate to 10.0.0.175, so you do not need the keys and configuration on bastion.
2. the one providing the Proxy, which only needs to be able to authenticate to bastion.

Your command for 1. works fine, but you are not seeing the debug output for 2. Since you used a resolvable TLD and everything else is common configuration, 2. is not spewing any errors - but its not doing what you want because it was never provided with the IdentityFile and User options from your config. It is authenticating with a different key or user, and rejected by bastion (rightfully so).

To ensure that 2. also reads your config, explicitly pass that option like this:

# Servers in availability zone A
Host 10.0.0.*
  ProxyCommand ssh -vvv -F /tmp/aws_bastion_ssh_config -W %h:%p bastion.example.com
  IdentityFile ~/.ssh/key.pem

# Servers in availability zone B
Host 10.0.1.*
  ProxyCommand ssh -vvv -F /tmp/aws_bastion_ssh_config -W %h:%p bastion.example.com
  IdentityFile ~/.ssh/key.pem

# The bastion host itself
Host bastion.example.com
  User ubuntu
  IdentityFile ~/.ssh/key.pem
  ControlMaster auto
  ControlPath ~/.ssh/ansible-%r@%h:%p
  ControlPersist 5m

And everything should work. Confirm with:

ssh -vvv -F /tmp/aws_bastion_ssh_config 10.0.0.175

Then remove the -vvv form both 1. and 2. If you move your /tmp/aws_bastion_ssh_config to the default location, both -F options can be removed (and ssh will read /etc/ssh/ssh_config and ~/.ssh/config files)