1

I have 2 clusters running in Azure for 2 different Availability Zones and I would like to cluster the etcd masters following https://kubernetes.io/docs/admin/high-availability/#replicated-api-servers .

I created the discovery token for 3 masters. When I try to init etcd container it fails:

2017-10-27 20:28:17.554393 I | etcdmain: etcd Version: 3.0.17
2017-10-27 20:28:17.554435 I | etcdmain: Git SHA: cc198e2
2017-10-27 20:28:17.554441 I | etcdmain: Go Version: go1.6.4
2017-10-27 20:28:17.554444 I | etcdmain: Go OS/Arch: linux/amd64
2017-10-27 20:28:17.554448 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2017-10-27 20:28:17.554511 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-10-27 20:28:17.554570 I | etcdmain: listening for peers on http://127.0.0.1:2380
2017-10-27 20:28:17.554594 I | etcdmain: listening for client requests on 127.0.0.1:4001
2017-10-27 20:28:17.868368 I | etcdmain: stopping listening for client requests on 127.0.0.1:4001
2017-10-27 20:28:17.868395 I | etcdmain: stopping listening for peers on http://127.0.0.1:2380
2017-10-27 20:28:17.868403 E | etcdmain: member "node-1" has previously registered with discovery service token (https://discovery.etcd.io/457f96956adb17ca0cc372e77b4e1420).
2017-10-27 20:28:17.868408 E | etcdmain: But etcd could not find valid cluster configuration in the given data dir (/var/etcd/data).
2017-10-27 20:28:17.868411 I | etcdmain: Please check the given data dir path if the previous bootstrap succeeded
2017-10-27 20:28:17.868462 I | etcdmain: or use a new discovery token if the previous bootstrap failed.

Both clusters were created by acs-engine in different AZ. Is that an issue related to having 2 different clusters?? If so, how can I do HA in this scenario ? I am trying that since Federation is not ready for Production yet.

I am totally locked on that.

I would really appreciate some help on this.

Thanks.

Hugo Marcelo Del Negro
  • 138
  • 1
  • 11

1 Answers1

0

Highly Available setups for Kubernetes masters assume you are running multiple (usually 3 so you can have a voting quorum) masters within the same cluster. Your current setup consists of 2 separate 1-master clusters.

When you have multiple clusters, you'll want to look at Cluster Federation although I'd wager this is not what you want, as you'd generally have federated clusters having 3 master setups each as well.

If you can't afford to destroy your existing clusters and boot them up already in HA mode, I suggest this excellent guide for migrating from single to multiple master setups as well as considering using kops for this type of operations.

vascop
  • 4,972
  • 4
  • 37
  • 50
  • Thanks a lot for looking on this. I must to perform all this on MS Azure so the only tool I have is acs-engine. I see kops is official for AWS which is not my scenario today. Now imagine I can increase the number of master nodes still. On the other hand my 2 k8s clusters runs on 2 different AZ. How can I do to reach a cross-region cluster master an synchronize all them? This is a production deployment, and Federation is not ready yet for that, neither for MS Azure. Do you believe its possible ? Do you know why discovery service dont recognize my other master in a different zone ? Tks! – Hugo Marcelo Del Negro Nov 03 '17 at 17:33
  • Is that the only way I have to increase the per-cluster number of instances to increase quorum of etcd nodes? – Hugo Marcelo Del Negro Nov 03 '17 at 17:35
  • The backend that API servers use is etcd and as such, the etcd nodes must be configured in a cluster themselves and not as 2 standalone nodes. – vascop Nov 03 '17 at 17:42