Can raft leader election algorithm run on serverless (lambda)?

Question

Is it possible to run Raft on serverless lambdas instead of a cluster? The main benefit would be not having to maintain a cluster or VMs.

Answer 1

(Disclaimer: though I read the original Raft paper years ago and have since refreshed myself using high-level articles, I'm not an expert on it, and have never implemented it.)

It may be possible, but you would have to work against the framework Lambda provides. You would be losing many of the benefits of both Lambda and Raft.

First, you'd need to figure out how to identify each instance (so that you can elect one as leader), and have that instance persist across Lambda invocations. A Lambda layer could probably do that, but it adds complexity.

Then, to elect a leader, you need a majority vote. A majority of... what? It would have to be the Lamdba's maximum concurrency, which each Lambda would need to query. Okay, not a huge deal; but it does mean that every time you have an election, you'd need at least $MaxConcurrency / 2 + 1 instances.

Then, you'd need to figure out how to heartbeat. That's not impossible, but it's pretty awkward to do in Lambdas; you'd probably need something like an SQS queue that you populate with $N messages, each of which triggers a heartbeat-receive. You need this for every follower — again, at least $MaxConcurrency / 2 + 1 — which means that you always have to have at least that many Lambdas active. At that point, you're basically using Lambda to run an expensive cluster.

Finally, you'd need a way to actually do something with the leader: how do you have it process requests? Lambdas can't open HTTP ports, so you can't use that; and you can't use API Gateway, since that will round-robin the requests to any ol' instance (rather than only to the leader). You could probably create an SQS queue that the Lambda manually polls from (using application code, not AWS's built-in triggers), but it'd be awkward. It's also not helpful: if what you want is an SQS queue that only one Lambda instance reads from, you can just create a FIFO queue triggering a Lambda with a max concurrency of 1.

In short, while it may be technically possible, you would probably have to run your Lambdas quite hot — half of them running 24/7 — which basically turns it into a very expensive cluster. In return, you'd have a leader which is at best difficult to actually use.

Answer 2

Raft requires a stable storage for state, so if your lambda is using ddb, s3 or any other persistent storage, then the raft logic itself may run in a lambda.

From design point of view, the code would have a storage abstraction (interface) with several implementations (e.g. one for testing, one for local, one for prod).

p.s. forgot to mention that lambdas will need some kind of concurrency control in place, so if a lambda X represents a raft node, no more than one instance of that lambda may modify the state