1

I have 3 servers and I want to connect them to build a k8s cluster over 10Gb NICs.

I am using this configuration right now.

server2 (10.0.0.2) <--> server1 <--> server3 (10.0.1.3)

how to connect 3 servers under two private networks

The problem with this config is that if server 1 is down, server2 and server3 cannot reach each other. And, if I add another cable between server2 and server3, it may solve the issue. But it seems each server will have at least 2 IPs (10.0.x.x). I have no idea how to tell k8s and ceph nodes to use fallback IP if one server is down.

So, is it possible to use "ip route" to let each server has an IP address that can be reached by each other without using a fallback ip after one server is down?

All 3 servers are in a datacenter. Adding a 10Gb switch is costly... It will be great if I don't need a switch.

Thanks

=======================================

Solution is found but not tested yet since I have not added the cable between server2 and server 3**

connection diagram

  • on server A

    route add -host 192.168.1.22 dev eth1

    route add -host 192.168.1.23 dev eth2

  • on server B

    route add -host 192.168.1.21 dev eth1

    route add -host 192.168.1.23 dev eth2

  • on server C

    route add -host 192.168.1.21 dev eth1

    route add -host 192.168.1.22 dev eth2

Source: https://www.cnblogs.com/zszmhd/p/3365161.html

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
Lod
  • 21
  • 6
  • In general a HA-cluster is of no use if it can’t communicate beyond the member nodes and you will still need a HA network connection in addition to those direct interconnects - so people generally make that network HA with redundant switches and hook up each server to two switches and once you have that HA network; why bother with direct connections – Bob Dec 28 '20 at 07:56
  • @HermanB They all have public IP addresses. I can use their public addresses to create the k8s cluster. But, the performance for the ceph storage will not be acceptable since the ports are less than 250 Mpbs. So, I have to use 10G direct connections. And, I don't worry about network stability. The only problem is I cannot power off one server to do some maintenance while keeping the private network connected. – Lod Dec 28 '20 at 08:56
  • Now you have three single points of failure. Eliminating them would require two switches. – Michael Hampton Dec 28 '20 at 18:15
  • @MichaelHampton They all have public addresses and any two servers can connect to each other directly through the private network. Anyone server goes down does not affect the other two servers. I don't see three single points of failure. – Lod Dec 30 '20 at 00:20

1 Answers1

0

  • on server A

    route add -host 192.168.1.22 dev eth1

    route add -host 192.168.1.23 dev eth2

  • on server B

    route add -host 192.168.1.21 dev eth1

    route add -host 192.168.1.23 dev eth2

  • on server C

    route add -host 192.168.1.21 dev eth1

    route add -host 192.168.1.22 dev eth2

Source: https://www.cnblogs.com/zszmhd/p/3365161.html

If you want to save the route info permanently, please read this.

how to save static routes permanently in ubuntu

Lod
  • 21
  • 6