We are building an application using the LMAX Disruptor. When using Event Sourcing, you often want to persist periodic snapshots of your domain model (some people call this the Memory Image pattern).
I need a better solution than what we are currently using to serialize our domain model when taking a snapshot. I want to be able to "pretty-print" this snapshot in a readable format for debugging, and I want to simplify snapshot schema migration.
Currently, we are using Googles' Protocol Buffers to serialize our domain model to a file. We chose this solution, because protocol buffers are more compact than XML / JSON, and using a compact binary format seemed like a good idea to serialize a big Java domain model.
The problem is, Protocol Buffers were designed for relatively small messages, and our domain model is quite big. So the domain model does not fit in one big hierarchical protobuf message, and we end up serializing various protobuf messages to a file, like this:
for each account {
write simple account fields (id, name, description) as one protobuf message
write number of user groups
for each user group {
convert user group to protobuf message, and serialize it
}
for each user {
convert user to protobuf message, and serialize it
}
for each sensor {
convert sensor to protobuf message, and serialize it
}
...
}
This is annoying, because manipulating a stream of heterogenous protobuf messages is complicated. It would be a lot easier if we had one big protobuf message that contained all of our domain model, like this:
public class AggregateRoot {
List<Account> accounts;
}
--> convert to big hierarchical protobuf message using some mapping code:
message AggregateRootMessage {
repeated AccountMessage accounts = 1;
}
--> persist this big message to a file
If we do this, it's easy to prettyprint a snapshot: simply read the big protobuf message, then prettyprint it using protobuf's TextFormat. With our current approach, we need to read the various protobuf messages one by one, and pretty-print them, which is harder, since the order of the protobuf messages in the stream depends on the current snapshot schema, so our pretty-printing tool needs to be aware of that.
I also need a tool to migrate snapshots to the new snapshot schema when our domain model evolves. I'm still working on this tool, but it's hard, because I have to deal with a stream of various protobuf messages, instead of dealing with just one big message. If it were just one big message, I could: - take the snapshot file - parse the file as a big Java protobuf message, using the .proto schema for the previous snapshot version - convert this big protobuf message into a big protobuf message for the new version, using Dozer and some mapping code - write this new protobuf message in a new file, using the .proto schema for the new version
But since I am dealing with a stream of protobuf messages of various types, my tool needs to handle this stream in the correct order.
So, yeah... I guess my questions are:
Do you know any serialization tool that can serialize a big domain model into a file, without protobuf's limitations, possibly using streaming to avoid OutOfMemorryErrors?
If you use event sourcing or memory images, what do you use to serialize your domain model? JSON? XML? Protobuf? Something else?
Are we doing it wrong? Do you have any suggestions?
