Ansible Playbook command timeout when connecting to cisco switch using SSH key

Question

Summary:

im trying to setup a playbook to get a list of IOS devices from netbox and then export the config files

Versions:

ansible-playbook [core 2.13.2]
python version = 3.8.13
switch IOS = 15.0(2)EX5
ansible host = Ubuntu 18.04.6 LTS

Issue:

When i run the command: ansible-playbook -vvvvv playbook.yml

I get the error:

The full traceback is:
  File "/usr/local/lib/python3.8/dist-packages/ansible_collections/cisco/ios/plugins/module_utils/network/ios/ios.py", line 94, in get_capabilities
    capabilities = Connection(module._socket_path).get_capabilities()
  File "/usr/local/lib/python3.8/dist-packages/ansible/module_utils/connection.py", line 200, in __rpc__
    raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
fatal: [PHC-16JUB-SW01]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "commands": [
                "show version"
            ],
            "interval": 1,
            "match": "all",
            "provider": null,
            "retries": 10,
            "wait_for": null
        }
    },
    "msg": "command timeout triggered, timeout value is 60 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."
}

Files:

playbook.yml

- name: Playbook to backup configs on Cisco Routers
  connection: network_cli
  hosts: device_roles_switch
  remote_user: admin
  gather_facts: False

  tasks:
    - name: Show Running configuration on Device
      ios_command:
        commands:
                - show version
      vars:
        ansible_user: admin
      register: config

    - name: Save output to backups folder
      copy:
        content: "{{ config.stdout[0] }}"
        dest: "./backups/{{ inventory_hostname }}-config.txt"

ansible.cfg

[defaults]
inventory = ./netbox_inv.yml
host_key_checking = false
retry_files_enabled = false
forks = 4
ansible_user = admin
remote_user = admin
private_key_file = ~/.ssh/id_rsa

[privilege_escalation]
become = True
become_method = enable
become_pass = passwordhere

[persistent_connection]
command_timeout = 60

netbox_inv.yml

plugin: netbox.netbox.nb_inventory
api_endpoint: http://172.16.1.32
token: 'apikeygoeshere'
validate_certs: False
config_context: False
group_by:
  - device_roles
query_filters:
  - tag: backup
  - status: active
compose:
  ansible_network_os: platform.slug

Troubleshooting:

I have confirmed i can use the key to directly connect to the host

root@netbox:~/config_backups# ssh admin@172.16.100.133
PHC-SW01>exit
Connection to 172.16.100.133 closed.

Confirmed SSH connection on switch:

PHC-SW01#debug ip ssh detail
ssh detail messages debugging is on
PHC-SW01#terminal monitor
PHC-SW01#
Aug  3 23:58:33.930: SSH1: starting SSH control process
Aug  3 23:58:33.930: SSH1: sent protocol version id SSH-1.99-Cisco-1.25
Aug  3 23:58:33.934: SSH1: protocol version id is - SSH-2.0-libssh_0.9.3
Aug  3 23:58:33.934: SSH2 1: SSH2_MSG_KEXINIT sent
Aug  3 23:58:33.934: SSH2 1: SSH2_MSG_KEXINIT received
Aug  3 23:58:33.934: SSH2:kex: client->server enc:aes256-cbc mac:hmac-sha1
Aug  3 23:58:33.934: SSH2:kex: server->client enc:aes256-cbc mac:hmac-sha1
Aug  3 23:58:34.004: SSH2 1: expecting SSH2_MSG_KEXDH_INIT
Aug  3 23:58:34.018: SSH2 1: SSH2_MSG_KEXDH_INIT received
Aug  3 23:58:34.147: SSH2: kex_derive_keys complete
Aug  3 23:58:34.150: SSH2 1: SSH2_MSG_NEWKEYS sent
Aug  3 23:58:34.150: SSH2 1: waiting for SSH2_MSG_NEWKEYS
Aug  3 23:58:34.154: SSH2 1: SSH2_MSG_NEWKEYS received
Aug  3 23:58:34.371: SSH2 1: Using method = none
Aug  3 23:58:34.378: SSH2 1: Using method = publickey
Aug  3 23:58:34.378: SSH2 1: Authenticating 'admin' with method: publickey
Aug  3 23:58:34.388: SSH2 1: Client Signature verification PASSED
Aug  3 23:58:34.388: SSH2 1: authentication successful for admin
Aug  3 23:58:34.392: SSH2 1: channel open request
Aug  3 23:58:34.395: SSH2 1: pty-req request
Aug  3 23:58:34.395: SSH2 1: setting TTY - requested: height 24, width 80; set:                                                                             height 24, width 80
Aug  3 23:58:34.395: SSH2 1: shell request
Aug  3 23:58:34.395: SSH2 1: shell message received
Aug  3 23:58:34.395: SSH2 1: starting shell for vty
Aug  3 23:58:34.409: SSH2 1: channel window adjust message received 1216017
Aug  3 23:59:04.035: SSH1: Session terminated normally

Also changed the timeout to 60 secs, no change even if the timeout is longer, appears it.

Log Output:

root@netbox:~/config_backups# ansible-playbook -vvvvv playbook.yml
ansible-playbook [core 2.13.2]
  config file = /root/config_backups/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.8.13 (default, Apr 19 2022, 00:53:22) [GCC 7.5.0]
  jinja version = 3.1.2
  libyaml = True
Using /root/config_backups/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /root/config_backups/netbox_inv.yml as it did not pass its verify_file() method
script declined parsing /root/config_backups/netbox_inv.yml as it did not pass its verify_file() method
Loading collection netbox.netbox from /root/.ansible/collections/ansible_collections/netbox/netbox
Using inventory plugin 'ansible_collections.netbox.netbox.plugins.inventory.nb_inventory' to process inventory source '/root/config_backups/netbox_inv.yml'
Fetching: http://172.16.1.32/api/status
Fetching: http://172.16.1.32/api/docs/?format=openapi
Fetching: http://172.16.1.32/api/dcim/devices/?limit=0&tag=backup&status=active&exclude=config_context
Fetching: http://172.16.1.32/api/virtualization/virtual-machines/?limit=0&tag=backup&status=active&exclude=config_context
Fetching: http://172.16.1.32/api/dcim/sites/?limit=0
Fetching: http://172.16.1.32/api/dcim/regions/?limit=0
Fetching: http://172.16.1.32/api/dcim/site-groups/?limit=0
Fetching: http://172.16.1.32/api/dcim/locations/?limit=0
Fetching: http://172.16.1.32/api/tenancy/tenants/?limit=0
Fetching: http://172.16.1.32/api/dcim/device-roles/?limit=0
Fetching: http://172.16.1.32/api/dcim/platforms/?limit=0
Fetching: http://172.16.1.32/api/dcim/device-types/?limit=0
Fetching: http://172.16.1.32/api/dcim/manufacturers/?limit=0
Fetching: http://172.16.1.32/api/virtualization/clusters/?limit=0
Fetching: http://172.16.1.32/api/ipam/services/?limit=0
Fetching: http://172.16.1.32/api/dcim/racks/?limit=0
Parsed /root/config_backups/netbox_inv.yml inventory source with auto plugin
[WARNING]: Skipping plugin (/usr/local/lib/python3.8/dist-packages/ansible/plugins/connection/winrm.py) as it seems to be invalid: invalid syntax (spawnbase.py, line 224)
redirecting (type: modules) ansible.builtin.ios_command to cisco.ios.ios_command
Loading collection cisco.ios from /usr/local/lib/python3.8/dist-packages/ansible_collections/cisco/ios
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.8/dist-packages/ansible/plugins/callback/default.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: playbook.yml **********************************************************************************************************************************************************************
Positional arguments: playbook.yml
verbosity: 5
private_key_file: /root/.ssh/id_rsa
remote_user: admin
connection: smart
timeout: 10
become: True
become_method: enable
tags: ('all',)
inventory: ('/root/config_backups/netbox_inv.yml',)
forks: 4
1 plays in playbook.yml

PLAY [Playbook to backup configs on Cisco Routers] ******************************************************************************************************************************************
META: ran handlers

TASK [Show Running configuration on Device] *************************************************************************************************************************************************
task path: /root/config_backups/playbook.yml:8
redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
Loading collection ansible.netcommon from /usr/local/lib/python3.8/dist-packages/ansible_collections/ansible/netcommon
redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
redirecting (type: become) ansible.builtin.enable to ansible.netcommon.enable
<172.16.9.1> attempting to start connection
<172.16.9.1> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /usr/local/bin/ansible-connection
<172.16.9.1> local domain socket does not exist, starting it
<172.16.9.1> control socket path is /root/.ansible/pc/4be7afdbe7
<172.16.9.1> redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
<172.16.9.1> Loading collection ansible.netcommon from /usr/local/lib/python3.8/dist-packages/ansible_collections/ansible/netcommon
<172.16.9.1> redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
<172.16.9.1> Loading collection cisco.ios from /usr/local/lib/python3.8/dist-packages/ansible_collections/cisco/ios
<172.16.9.1> redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<172.16.9.1> local domain socket listeners started successfully
<172.16.9.1> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /usr/local/lib/python3.8/dist-packages/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os ios
<172.16.9.1> ssh type is set to auto
<172.16.9.1> autodetecting ssh_type
<172.16.9.1> ssh type is now set to libssh
<172.16.9.1>
<172.16.9.1> local domain socket path is /root/.ansible/pc/4be7afdbe7
redirecting (type: modules) ansible.builtin.ios_command to cisco.ios.ios_command
redirecting (type: action) ansible.builtin.ios to cisco.ios.ios
<172.16.9.1> Using network group action ios for ios_command
redirecting (type: action) ansible.builtin.ios to cisco.ios.ios
<172.16.9.1> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
redirecting (type: modules) ansible.builtin.ios_command to cisco.ios.ios_command
<172.16.9.1> ANSIBLE_NETWORK_IMPORT_MODULES: found ios_command  at /usr/local/lib/python3.8/dist-packages/ansible_collections/cisco/ios/plugins/modules/ios_command.py
<172.16.9.1> ANSIBLE_NETWORK_IMPORT_MODULES: running ios_command
<172.16.9.1> ANSIBLE_NETWORK_IMPORT_MODULES: complete
<172.16.9.1> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, 'msg': 'command timeout triggered, timeout value is 30 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide.', 'exception': '  File "/usr/local/lib/python3.8/dist-packages/ansible_collections/cisco/ios/plugins/module_utils/network/ios/ios.py", line 94, in get_capabilities\n    capabilities = Connection(module._socket_path).get_capabilities()\n  File "/usr/local/lib/python3.8/dist-packages/ansible/module_utils/connection.py", line 200, in __rpc__\n    raise ConnectionError(to_text(msg, errors=\'surrogate_then_replace\'), code=code)\n', 'invocation': {'module_args': {'commands': ['show version'], 'match': 'all', 'retries': 10, 'interval': 1, 'wait_for': None, 'provider': None}}, '_ansible_parsed': True}
The full traceback is:
  File "/usr/local/lib/python3.8/dist-packages/ansible_collections/cisco/ios/plugins/module_utils/network/ios/ios.py", line 94, in get_capabilities
    capabilities = Connection(module._socket_path).get_capabilities()
  File "/usr/local/lib/python3.8/dist-packages/ansible/module_utils/connection.py", line 200, in __rpc__
    raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
fatal: [PHC-SW01]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "commands": [
                "show version"
            ],
            "interval": 1,
            "match": "all",
            "provider": null,
            "retries": 10,
            "wait_for": null
        }
    },
    "msg": "command timeout triggered, timeout value is 30 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."
}

PLAY RECAP **********************************************************************************************************************************************************************************
PHC-SW01             : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

^{<172.16.9.1> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, 'msg': 'command timeout triggered, timeout value is 30 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide.', 'exception': ' File "/usr/local/lib/python3.8/dist-packages/ansible_collections/cisco/ios/plugins/module_utils/network/ios/ios.py", line 94, in get_capabilities\n capabilities = Connection(module._socket_path).get_capabilities()\n File "/usr/local/lib/python3.8/dist-packages/ansible/module_utils/connection.py", line 200, in rpc\n raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)\n', 'invocation': {'module_args': {'commands': ['show version'], 'match': 'all', 'retries': 10, 'interval': 1, 'wait_for': None, 'provider': None}}, '_ansible_parsed': True}}

Im not sure if the issue is with authentication, or the command being run. the command show version doesnt need privilage escalation.

UPDATE

I Enabled Logging and got the below;

2 things i noticed was;

command: None ? command was show version

Response 2 ? privilage escalation enacted 'enable' did this work?

 | invoked shell using ssh_type: libssh
 | ssh connection done, setting terminal
 | loaded terminal plugin for network_os ios
 | command: None
 | response-1: b'\r\nPHC-SW01>'
 | matched cli prompt 'b'\nPHC-SW01>'' with regex 'b'[\\r\\n]?[\\w\\+\\-\\.:\\/\\[\\]]+(?:\\([^\\)]+\\)){0,3}(?:[>#]) ?$'' from response 'b'\r\nPHC-SW01>''
 | firing event: on_become
 | send command: b'enable\r'
 | command: b'enable'
 | response-1: b'enabl'
 | response-2: b'e\r\nPassword: '
 | resetting persistent connection for socket_path /root/.ansible/pc/31759300a1
 | closing ssh connection to device

Have you tried using -u to specify the user when running the playbook like in the example below? ansible-playbook -u admin -i netbox_inv.yml playbook.yml — , Aug 03 '22 at 05:43
@Ivan Krizsan. Both `remote_user` and `ansible_user` default to `admin` in `ansible.cfg` — Vladimir Botka, Aug 03 '22 at 05:48
OK. The [issues](https://github.com/ansible-collections/cisco.ios/issues?q=is%3Aissue+surrogate_then_replace) might be relevant. — Vladimir Botka, Aug 03 '22 at 06:38
I enabled logging on SSH and confirmed its connecting, just the command not executing [pastebin](https://pastebin.com/LcEwa1bg) I checked the previous issues and only thing close to mine is [Issue 369](https://github.com/ansible-collections/cisco.ios/issues/369) i even set the [playbook.yml](https://pastebin.com/wX1kS9qh) to display the output when running to see what might cause it but same timeout issue, its like it connects and waits for something but i cant work out how to see what its waiting for? direct SSH shows no prompt for the command show version — DrunkMunki, Aug 04 '22 at 00:14