0

ceph does not seem to be very resilient to network moves and changes. Perhaps because the IP address is stored as an identification of the systems that make up the ceph cluster.

The problem: recently our cluster was physically moved with very little warning, ceph was closed down and the systems were shutdown just prior to the move. The new location had different IP address allocation and ceph did not boot correctly when the cluster was powered back on and was allocated new IP addresses.

Possible solutions: Avoid persistent store of IP address and host name. Instead, when the ceph cluster is booted, the association between host name and physical devices should be made at run time.

If IP address and host name have to be kept in persistent store there should be a tool to enable easy migration between different network configurations.

More on the failure case The ceph-mon.*.asok sockets did not exist because the monitor service would not start. The errors preventing start of the monitor appeared to be that the monitor could not connect to the cluster - because as stated our cluster was shut down quickly and the IP addresses were not change in a controlled way. The monitor log is full of the attempts to start with the following bind error: Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address

Things that were tried to migrate to the new network configuration Starting point was to update ceph.conf with the new IP address configuration and to re-run ceph-deploy admin (the ceph version is a Nautilus) to push the new admin configuration to the network. This however does not update the ceph crush maps. Various old discussions for example https://tracker.ceph.com/issues/3550 were studied and the user documents https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address, but no real workable solution for this was found.

So I am asking this question to see if anyone else has had the same problem and if so, what the real experts think about only associating host name / IP address and physical devices at run time and implementing a discovery stage during ceph boot where the association is made. Or of course perhaps there is another better solution?

Pete
  • 143
  • 8
  • There have been several threads in the ceph-users mailing list describing how to move a ceph cluster to a new ip address range. You'll have to edit the monmap (you already found that doc) and then change `public address` (and `cluster address` if they're separated). Which described way did you apply to change the address ("right" or "messy" way)? – eblock Nov 05 '20 at 06:56
  • @eblock This is actually the answer. Changing the IP of the monitors without updating the monmap is the problem. The docs are displaying various solutions at https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address – itsafire Nov 05 '20 at 08:20
  • I have updated my description. The monitor would not start so I could not use the instructions to save the monitor map and update the IP addresses. In the end I gave up and lost the data to get the ceph cluster back running again. As part of that I discovered for some reason that the cephx authentication was also preventing the ceph -s command work so I switched that off as well. The whole experience was very unsatisfactory and could have been avoided (in my opinion) if https://man7.org/linux/man-pages/man3/getifaddrs.3.html had been used in the code. – Pete Nov 06 '20 at 08:32

0 Answers0