0

I recently configured VLAN interfaces on my relatively new Ubuntu 18.04 server which has two physical interfaces: eno1 and eno2. I have configured them using the netplan utility.

My goal is simple: to be able to SSH from the gateway (a pair of Cisco ASR 1006's running HSRP). In other words, I'm simply trying to SSH from host to host within the same LAN. I have tried everything I can think of to troubleshoot this. Here are some notes:

  • SSH works from the Ubuntu's VLAN interfaces to the gateway
  • SSH works from the Cisco ASR to the IP address of the physical interface of the server
  • SSH does not work from the Cisco ASR to the IP address of the VLAN interface on the server
  • Packet captures from the Cisco ASR side show the SSH request being sent to the server, but the server doesn't reply so a TCP retransmit goes out a few times before the ASR gives up.
  • Packet captures from the Ubuntu server show the same thing. The packets come in on TCP port 22 from the host trying to SSH (the ASR 1006), but the server does not respond, so TCP retransmits come in next.
  • I've reinstalled open-ssh server, rebooted, restarted services, explicitly specified sshd to listen on 10.255.255.12, etc, to no avail.
  • A normal host (Win 8.1, using PuTTY) can ssh into the vlan interface of the server.

The problem: The Cisco ASR cannot SSH into the vlan interface of the server.

VLANs 999 and 3001 contain a private IP and a private CGN IP respectively:

Contents of /etc/netplan/50-cloud-init.yaml

# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        eno1:
            addresses:
            - <my public IP>/28
            gateway4: <my gateway>
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
        eno2:
            dhcp4: true
        enp10s0f0:
            dhcp4: true
        enp10s0f1:
            dhcp4: true
    version: 2

    vlans:
        vlan.3001:
            id: 3001
            link: eno2
            addresses: [10.255.255.12/29]
        vlan.999:
            id: 999
            link: eno2
            addresses: [100.78.32.240/24]

Contents of /etc/ssh/sshd_config

#       $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
ListenAddress <my public ip>
ListenAddress 10.255.255.12
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile     .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem sftp  /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server
PasswordAuthentication yes

The sshd service is apparently listening on the private IP interface, but still won't reply to ssh requests from the ASR to its VLAN interfaces.

~$ sudo netstat -tulpn | grep :22
tcp        0      0 <my public IP>:22       0.0.0.0:*               LISTEN      3956/sshd           
tcp        0      0 10.255.255.12:22        0.0.0.0:*               LISTEN      3956/sshd

Note that this is from the server itself (SSH’ing to self):

$ ssh 10.255.255.12
<my username>@10.255.255.12's password:

Note that this is from the Cisco ASR (the -l flag is simply the username to use):

ASR1006#ssh -l user <my public IP>
Password: 
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-72-generic x86_64)

user@ubuntu-server:~$ exit
logout


[Connection to <my public IP> closed by foreign host]
ASR1006#ssh -l user 100.78.32.240
% Connection timed out; remote host not responding


ASR1006#ssh -l user 10.255.255.12
% Connection timed out; remote host not responding

You can see that the connection was established when SSH'ing to the public IP address associated with the server's physical interface: eno1.

The final part of my troubleshooting involved connecting a normal host - Windows 8.1 using PuTTY, and putting it on the 10.255.255.8/29 subnet. The Windows host could SSH into the server's vlan.3001 interface.

This is such a unique problem that I'm not optimistic about anyone being able to assist. I can open a TAC case with Cisco since the problem could be on the Cisco side, but I wanted to start here.

David00
  • 1
  • 2
  • You could use tcpdump on eno2, specifically with the -e option to display VLANs, to check if VLAN IDs are set as expected when receiving and sending traffic. It's possible for example that you have a routing problem with regard to the IP used by the switch (what IP is it?). That's why using tcpdump -e to look for anomalies seems the way to go for now. – A.B Dec 28 '19 at 16:25
  • @A.B - I've accomplished the same task by using the -w flag and exporting the tcpdump to wireshark for analysis. I can confirm that the SSH requests are encapsulated as expected. Also, I sent some pings from the server to the default gateway - they are encapsulated in the correct VLAN as well. – David00 Dec 28 '19 at 21:23

0 Answers0