1

I would like to use Apache Marathon to manage resources in a clustered product. Mesos and Marathon solves some of the "cluster resource manager" problems for additional components that need to be kept running with HA, failover, etc.

However, there are a number of services that need to be kept running to keep mesos and marathon running (like zookeeper, mesos itself, etc). What can we use to keep those services running with HA, failover, etc?

It seems like solving this across a cluster (managing how many instances of zookeeper, etc, and where they run and how they fail over) is exactly the problem that mesos/marathon are trying to solve.

TREE
  • 1,292
  • 2
  • 12
  • 21

2 Answers2

3

As the Mesos HA doc explains, you can start multiple Mesos masters and let ZK elect the leader. Then if your leading master fails, you still have at least 2 left to handle things. It is common to use something like systemd to automatically restart the mesos-master on the same host if it's still healthy, or something like Amazon AutoScalingGroups to ensure you always have 3 master machines even if a host dies.

The same can be done for Marathon in its HA mode (on by default if you start multiple instances pointing to the same znode). Many users start these on the same 3 nodes as their Mesos masters, using systemd to restart failed Marathon services, and the same ASG to ensure there are 3 Mesos/Marathon master nodes.

These same 3 nodes are often configured to be the ZK quorum as well, so there are only 3 nodes you have to manage for all these services running outside of Mesos.

Conceivably, you could bootstrap both Mesos-master and Marathon into the cluster as Marathon/Mesos tasks. Spin up a single Mesos+Marathon master to get the cluster started, then create a Mesos-master app in Marathon to launch 2-3 masters as Mesos tasks, and a Marathon-master app in Marathon to launch a couple of HA Marathon instances (as Mesos tasks). Once those are healthy, you can kill the original standalone Mesos/Marathon master and the cluster would failover to the self-hosted Mesos and Marathon masters, which would be automatically restarted elsewhere on the cluster if they failed. Maybe this would work with ZK too. You'd probably need something like Mesos-DNS and/or ELB to let other services find Mesos/Marathon. I doubt anybody's running Mesos this way, but it's crazy enough it just might work!

Adam
  • 4,322
  • 1
  • 16
  • 22
  • So, on Amazon, you could use AutoScalingGroups? That keeps individual processes running? What about on regular old linux boxes? – TREE Apr 06 '15 at 11:38
  • Systemd could definitely help with this if it were only a single node. But there may be an arbitrary number of nodes involved once migration (to replace a permanently failed master) is taken into account. – TREE Apr 06 '15 at 11:46
0

In order to understand this, I suggest you spend a few minutes reading up on the architecture and the HA part in the official Mesos doc. There, it is clearly explained how HA/failover in Mesos core is handled (which is, BTW, nothing magic—many systems I know of use pretty much exactly this model, incl. HBase, Storm, Kafka, etc.).

Also, note that—naturally—the challenge keeping a handful of the Mesos masters/Zk alive is not directly comparable with keeping potentially 10000s of processes across a cluster alive, evict them or fail them over (in terms of fan out, memory footprint, throughput, etc.).

Michael Hausenblas
  • 13,162
  • 4
  • 52
  • 66
  • How is keeping Mesos master, ZK, and Marathon running with their respective state any different from running any other distributed service? You have to manage which nodes are running those services, Keep the state around, and be able to migrate them to other nodes in case of failure, including letting other services Discover where they went. The question is, what tools exist to help with that cluster management? – TREE Apr 06 '15 at 11:43
  • I'd argue that it's different in terms of scale: in the case of 3 Zk or Mesos masters you can afford certain techniques. You can however not when, for example, you try to ensure thousands of scan tasks in a multi-level execution tree keep running (Dremel, http://research.google.com/pubs/pub36632.html). Another example is 'heart-beat overhead', see e.g. HDFS findings in https://www.usenix.org/legacy/publications/login/2010-04/openpdfs/shvachko.pdf or the # of ports/threads necessary just to keep up the communication. – Michael Hausenblas Apr 11 '15 at 04:49