I am implementing data aggregation application framework. It should be consuming create, update and delete operations on MySQL databases which are sharded across multiple instances. These aggregations should be stored in a separate database and serve as read store for eg. statistical overviews.
I've come up with two solutions, one of which is using Kafka consumers to write this data to aggregation database. Other one is creating background jobs for Beanstalkd on application level (whenever data is persisted) and consuming them with workers which would do the same job as Kafka consumers.
I'm interested in your thoughts on in pros and cons of both approaches in terms of performance, scalability and fault tolerance.
