Is serverless in combination with event sourcing possible?

Question

I've been wanting to implement a mobile app using a complete serverless architecture for some time now and finally started looking into the details. So far, I've discovered that AWS offers most of the services that would be required for such a setup (API Gateway, Cognito, Lambda, DynamoDB, SQS, etc.), but I've yet to solve one (possibly theoretical) issue; event sourcing.

Since (historical) data is becoming more and more valuable these days, it's very important (in my humble opinion) to store historical data about your users. Current event sourcing offerings such as Akka Persistence achieve this goal by only persisting the events to a database and keeping the current state in memory (and saving snapshots to a database, etc.).

My problem with this, is that I don't have the ability to store such a state in memory since my Lambda functions terminate after their single purpose has been fulfilled. What my question boils down to, is there a framework currently out there that would support event sourcing (on Java) that saves the current state in something like ElastiCache (Redis). Since I have a lot of experience with Akka, is this something Persistence can already do? Is it worth chasing event sourcing in combination with a serverless backend (at this time) or is it simply not yet the right time this?

I haven't been able to find much in the Akka Persistence docs as of yet about this (possible non) issue. Please give me suggestions as to what I might have missed in my mission to a serverless universe; I'm still learning, as we all are.

Answer 1

This will be primarily opinion-based, so not the best fit for Stack Overflow, but I'll try to stay as factual as possible.

akka-persistence does not lend itself well to serverless deployment strategy, for the following reason. It relies on the strong assumption that at any one time there is only one PersistentActor for a given id. In a distributed environment enforcing this implies inter-node coordination, typically using akka-cluster-sharding. This doesn't yield itself to being deployed in a serverless environment that is meant to run simple functions.

In general, event sourcing implies rebuilding state from events stored in the journal (or latest snapshot + the events that followed) and doing that on top of a stateless environment means a lot of inefficiency for every single execution of the function, since there can be no local cache. Adding a distributed cache on top of event sourcing can mitigate that somewhat. However, you're still left with the challenge of coordination to prevent race-conditions between multiple instances of the function. These factors work against the operational simplicity that serverless is meant to provide.

Answer 2

Yes, you can do event-sourcing in Serverless.

One approach using AWS is to use DynamoDB as your Event Store. You can then use DynamoDB Streams with Lambda triggers to materialize them into your State Store (which can be any other DB).

Answer 3

You can do it with DynamoDB Streams, but only an Event Store is not enough. The code that generates the next event should be serialised, i.e. only one code instance at the time is allowed produce an event for a particular aggregate instance. Otherwise the order of the events may not be certain.

With event sourcing commands are sent to aggregates. When a command has an effect, i.e. modifies the aggregate, an event is generated, added to the log and usually published. Very often producing an event for a command requires the current state of the aggregate. This is why such code should not run in parallel for the same aggregate instance.

A solution is to have a “Command Store”, which is a DynamoDB table that stores the last command for each aggregate instance. Thus, the associated stream consists of updates to that item. The Lambda trigger for this stream reconstructs the state of the aggregate instance using the Event Store and generates the new event. The event is then saved in the Event Store. The stream of the event store takes care of the publication of the event.

In order to speed up the reconstruction of the state of an aggregate, a snapshot table can be used. Every 100 events, for example, the complete aggregate can be updated in it. Reconstruction then consists of fetching the snapshot and then fetching only the events with a sequence number above the one in the snapshot.

Numbering the events and the associated aggregate copies, which may exist in various Read Stores, has the advantage of making idempotence easy. This way replay of events is possible.

Answer 4

You can do event-sourcing using akka-persistence in a Lambda, if you're fine with (some configurable) eventual consistency and willing to also apply CQRS.

How does such a setup look like?

You will have (1 or more) lambda functions that create n lambda instances (lets call them QUERY-Lambda; where n is basically unlimited or limited by the concurrency limit available in your account) that are able to handle your read-sides (handling queries by reading the journal/snapshot store and then answering), and max. 1 instance of the lambda per aggregate that is handling write-operations (ensure this using the concurrency parameter in the lambda's configuration) (lets call them COMMAND-Lambda). This is important to make sure that the journal does not get corrupt by having more than one actor writing to it.

Depending on your consistency guarantees, make sure to stop the actors in the QUERY-Lambdas either right after the query is handled or set the receive timeout to a value that is fine with your consistency guarantee, knowing that multiple actors could give you a different state.

If you have CRUD operations, make sure that Read operations that show the user the current state before applying a change (e.g. showing the current values of a customer object in a form before updating it) are also handled by the WRITE-Lambda so you are sure that the state you are changing is the last state available.

You don't have to create multiple jar-files for this, you can simply deploy the same jar file as multiple lambda functions. You need to make sure in your API Gateway, that the requests changing state are routed to the WRITE-Lambda(s) and those where consistency is not so important are routed to the READ-Lambda(s).

Also make sure that you're not creating snapshots while replaying the journal, but only when doing command processing (because the READ-Lambdas are also replaying the journal and thereby could corrupt your state if they would create snapshots)

For best performance, create a snapshot after every command that changed state or at least before shutting down the actor, that way the next invocations will have to do a minimal amount of reads. AFAIK the Lambdas in Java also stay active for a reasonable time, so the cold-restarts should not be that much of a problem. If they are for u, create some cron that every ~5-10 minutes calls into the lambda to keep it alive. Alternatively you can use https://doc.akka.io/docs/alpakka/current/awslambda.html to simply send a request to yourself every x minutes.. You can use Source.tick(3 minutes) and then just invoke your WRITE-Lambda function like its shown in the Alpakka documentation.

Also make sure that operations where you'd need to talk with two aggregates (Saga / Coordinator) are handled by the same WRITE-Lambda. This might become a bottleneck, but of course you still can apply some kind of sharding through routing in the API Gateway. Its just more effort than if you would have a normal Akka Cluster.

If something is unclear, please leave a comment and I'll try to answer.