I have 3 servers:
- Server #1 runs among other things a MongoDB database
- Server #2 runs some apps that connect to server #1's mongodb, everything works fine
- Server #3 is a new server where more apps will connect to server #1's mongodb
My issue is with server #3, i am trying to get the app to connect to server #1's mongo without success. I've been looking at server #2 for reference and trying to replicate the configs there but somehow it seems i am missing something.
initially i noticed the firewall was up so not even incoming HTTP traffic was allowed, so i disabled UFW:
sudo ufw disable
sudo ufw status verbose
Status: inactive
Then i noticed that there was no route to reach server #1, which has an LAN IP of 10.100.116.65, as i couldn't ping it:
sudo route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 [GATEWAY-IP] 0.0.0.0 UG 0 0 0 enp1s0f0
[SERVER-IP] 0.0.0.0 255.255.255.248 U 0 0 0 enp1s0f0
I tried to add the route but the device was "not up":
sudo ip route add 10.100.116.0/24 via 0.0.0.0 dev enp1s0f1
Error: Device for nexthop is not up.
So i checked:
ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:18:64:58 brd ff:ff:ff:ff:ff:ff
3: enp1s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:18:64:59 brd ff:ff:ff:ff:ff:ff
Then upped it:
sudo ifconfig enp1s0f1 up
ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:18:64:58 brd ff:ff:ff:ff:ff:ff
3: enp1s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:18:64:59 brd ff:ff:ff:ff:ff:ff
Now that it's up, i added a route to reach server #1:
sudo ip route add 10.100.116.0/24 via 0.0.0.0 dev enp1s0f1
sudo route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 [GATEWAY-IP] 0.0.0.0 UG 0 0 0 enp1s0f0
10.100.116.0 0.0.0.0 255.255.255.0 U 0 0 0 enp1s0f1
[SERVER-IP] 0.0.0.0 255.255.255.248 U 0 0 0 enp1s0f0
But i still couldn't ping server #1:
ping 10.100.116.65
PING 10.100.116.65 (10.100.116.65) 56(84) bytes of data.
From X.X.X.X icmp_seq=1 Destination Host Unreachable
From X.X.X.X icmp_seq=2 Destination Host Unreachable
From X.X.X.X icmp_seq=3 Destination Host Unreachable
From X.X.X.X icmp_seq=4 Destination Host Unreachable
^C
--- 10.100.116.65 ping statistics ---
6 packets transmitted, 0 received, +4 errors, 100% packet loss, time 5102ms
I kept comparing Server #2's configs with server #3's and noticed that server #3 didn't have a LAN IP assigned yet for enp1s0f1:
ifconfig
enp1s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet X.X.X.X netmask 255.255.255.248 broadcast X.X.X.X
inet6 fe80::ec4:7aff:fe18:6458 prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:18:64:58 txqueuelen 1000 (Ethernet)
RX packets 7470666 bytes 2108978738 (2.1 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8563092 bytes 6460993288 (6.4 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xe0000000-e007ffff
enp1s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::ec4:7aff:fe18:6459 prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:18:64:59 txqueuelen 1000 (Ethernet)
RX packets 125640 bytes 25067378 (25.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1630 bytes 92016 (92.0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xe0080000-e00fffff
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 4317856 bytes 524128762 (524.1 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4317856 bytes 524128762 (524.1 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
So i added one in the netplan file:
sudo pico /etc/netplan/01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
id0:
match:
macaddress: 0c:c4:7a:18:64:58
addresses: [X.X.X.X/29]
gateway4: X.X.X.X
nameservers:
addresses: [8.8.8.8,8.8.4.4]
# i added these 2 lines
enp1s0f1:
addresses: [ 10.100.116.62/24 ]
Then saved and applied with sudo netplan apply, confirmed with ifconfig:
ifconfig
enp1s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet X.X.X.X netmask 255.255.255.248 broadcast X.X.X.X
inet6 fe80::ec4:7aff:fe18:6458 prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:18:64:58 txqueuelen 1000 (Ethernet)
RX packets 7471402 bytes 2109059743 (2.1 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8563748 bytes 6461111312 (6.4 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xe0000000-e007ffff
enp1s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.100.116.62 netmask 255.255.255.0 broadcast 10.100.116.255
inet6 fe80::ec4:7aff:fe18:6459 prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:18:64:59 txqueuelen 1000 (Ethernet)
RX packets 149491 bytes 29787304 (29.7 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1975 bytes 111102 (111.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xe0080000-e00fffff
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 4331691 bytes 525564926 (525.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4331691 bytes 525564926 (525.5 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Finally some success, i can now ping server #1:
ping 10.100.116.65
PING 10.100.116.65 (10.100.116.65) 56(84) bytes of data.
64 bytes from 10.100.116.65: icmp_seq=1 ttl=64 time=0.094 ms
64 bytes from 10.100.116.65: icmp_seq=2 ttl=64 time=0.254 ms
64 bytes from 10.100.116.65: icmp_seq=3 ttl=64 time=0.221 ms
64 bytes from 10.100.116.65: icmp_seq=4 ttl=64 time=0.245 ms
64 bytes from 10.100.116.65: icmp_seq=5 ttl=64 time=0.248 ms
^C
--- 10.100.116.65 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4096ms
rtt min/avg/max/mdev = 0.094/0.212/0.254/0.061 ms
And yet, event after restarting the app, it throws an error when trying to connect to server #1's mongo complaining there is no route to host:
MONGODB | Error checking 10.100.116.65:27017: Mongo::Error::SocketError: Errno::EHOSTUNREACH: No route to host - connect(2) for 10.100.116.65:27017 (for 10.100.116.65:27017 (no TLS)) (on 10.100.116.65:27017)
I've even rebooted the server just in case but that didn't help.
I'm a developer so i'm a bit clueless about all this so i'm probably missing something obvious but i have no idea what, can anybody shed some light on this?
EDIT
I've also looked at the iptables for server #1, it had a rule to accept traffic from server #2 so i also added the same rule for server #3:
sudo iptables -A INPUT -s 10.100.116.62/32 -m comment --comment server3 -j ACCEPT
i've also added a rule in server #3's iptables to accept traffic from server #1:
sudo iptables -A INPUT -s 10.100.116.65/32 -m comment --comment server1 -j ACCEPT
But i'm still getting the same error.
