0

Our CentOS 8 server at Hetzner has a primary IP aquired through DHCP, and an additional /29 subnet. No matter what outgoing IP I choose, packets are always sent with the primary IP. I would like to choose different outgoing IPs for different services, i.e. DNS and SMTP.

Anonymized setup:

  • Primary IP: 1.0.0.213/27
  • Secondary subnet: 2.0.0.88/29
  • Test IP: 2.0.0.89
  • NIC: enp3s0

Add IP secondary subnet:

$ nmcli con modify enp3s0 ipv4.addresses "2.0.0.89/32"
$ reboot

Configuration:

$ ip addr
1: lo: ...
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 2.0.0.89/32 scope global noprefixroute enp3s0
       valid_lft forever preferred_lft forever
    inet 1.0.0.213/27 brd 1.0.0.223 scope global dynamic noprefixroute enp3s0
       valid_lft 43105sec preferred_lft 43105sec
    inet6 ...
    inet6 ...

$ ip route
default via 1.0.0.193 dev enp3s0 proto dhcp metric 100 
1.0.0.192/27 dev enp3s0 proto kernel scope link src 1.0.0.213 metric 100

Test with our own what's-my-ip service hosted elsewhere:

$ curl -v -v -v -v -v --silent --interface 2.0.0.89 ip.mycompany.tld 
* Rebuilt URL to: ip.mycompany.tld/
*   Trying XX.XX.XX.XX...
* TCP_NODELAY set
* Name '2.0.0.89' family 2 resolved to '2.0.0.89' family 2 <-- correctly bound to 2.0.0.89?
* Local port: 0
* Connected to ip.mycompany.tld (XX.XX.XX.XX) port 80 (#0)
> GET / HTTP/1.1
...
1.0.0.213 <-- Expected 2.0.0.89 here?!?

Anonymized tcpdump confirms we're using the primary IP:

16:29:57.850153 IP (tos 0x0, ttl 64, id 4961, offset 0, flags [DF], proto TCP (6), length 60)                                                                   
    1.0.0.213.47307 > XX.XX.XX.XX.http: Flags [S], cksum 0xcae8 (incorrect -> 0x3f22), seq 1180758295, win 29200, options [mss 1460,sackOK,TS val 4199670061 ecr
 0,nop,wscale 7], length 0                                                                                                                                      
16:29:57.861321 IP (tos 0x0, ttl 57, id 0, offset 0, flags [DF], proto TCP (6), length 60)                                                                      
    XX.XX.XX.XX.http > 1.0.0.213.47307: Flags [S.], cksum 0xe167 (correct), seq 1203046991, ack 1180758296, win 28960, options [mss 1460,sackOK,TS val 255416431
1 ecr 4199670061,nop,wscale 7], length 0                                                                                                                        
16:29:57.861374 IP (tos 0x0, ttl 64, id 4962, offset 0, flags [DF], proto TCP (6), length 52)                                                                   
    1.0.0.213.47307 > XX.XX.XX.XX.http: Flags [.], cksum 0xcae0 (incorrect -> 0x8064), seq 1, ack 1, win 229, options [nop,nop,TS val 4199670072 ecr 2554164311]
, length 0                                                                                                                                                      
16:29:57.861468 IP (tos 0x0, ttl 64, id 4963, offset 0, flags [DF], proto TCP (6), length 133)                                                                  
    1.0.0.213.47307 > XX.XX.XX.XX.http: Flags [P.], cksum 0xcb31 (incorrect -> 0xf6dc), seq 1:82, ack 1, win 229, options [nop,nop,TS val 4199670072 ecr 2554164
311], length 81: HTTP, length: 81

Hetzner's support page on additional IPs: https://docs.hetzner.com/robot/dedicated-server/ip/additional-ip-adresses/

What am I missing here? :-)

  • CentOS 8.2.2004
  • Firewalld + NetworkManager
  • SELinux enabled
Saustrup
  • 1,183
  • 1
  • 8
  • 12
  • While NOT the proper soluton, I - out of pure desperation - tried to add a SNAT rule for outgoing HTTP requests, and that worked - so now I at least know that outgoing traffic on one of the alternate IPs is possible. What I tested: iptables -t nat -A POSTROUTING -p tcp --dport 80 -o enp3s0 -j SNAT --to-source 2.0.0.89 => HTTP request to our IP service returned 2.0.0.89 as the remote IP. – Saustrup Sep 11 '20 at 15:55

2 Answers2

0

You need to change your default route to the IP address you want to use. Your output shows 1.0.0.213 is the current default. Something like:

ip route replace default via 1.0.0.193 dev eth0 src 2.0.0.89
Bert
  • 2,863
  • 12
  • 13
  • I've tried to remove all other routes except the one you suggested, "default via 1.0.0.193 dev enp3s0 src 2.0.0.89 metric 100", but the detected outgoing address is still 1.0.0.213 - it's mindboggling. I've added Hetzner's documentation on this to the question, but their advice doesn't seem to work either. – Saustrup Sep 11 '20 at 14:30
0

Apparently this happened because I enabled masquerading on the public zone, and forgotten all about it. The reason I did this, was to allow the Docker containers to reach the internet. Apparently this will cause all non-primary IPs to masquerade as the primary IP, even if they're listed on the interface.

Turning off masquerading and rebooting fixed the issue:

$ firewall-cmd --remove-masquerade --zone=public --permanent
$ reboot

This does leave me with a container connectivity issue, but that could probably be fixed with a more precise SNAT/masquerade rule.

Saustrup
  • 1,183
  • 1
  • 8
  • 12