We have a system that sends us events when any attribute has changed on an order. They have a full order structure with say 500 attributes and we only storing a subset of them say 50 attributes + some transformation to our internal order data model. Now whenever any of these 500 attributes on the order changes at the source system, it triggers an event to us (note that they share the complete order state every time). Most of the times nothing changes on the 50 attributes we have in our model so many are duplicate events for us. Currently we are storing all the events we receive from the source in a database.
The requirement now is that we have to generate events to other consumers when the order data in our system has changed. So we will have to eliminate the duplicates from the source and only share the event when some attribute on our model has changed.
Possible solution: For every event we receive from source we could compare it with our internal data if something has changed on "any" attribute. We could have a memory database like REDIS. But there is going to be a lot of IO on the database. Also not sure whats that best way to compare if 2 json objects are exactly the same. Other solution is that, instead of storing all attributes we could store only some important "meta data" and compare on that meta data only. Will be good for performance but will not be fully perfect if we have to generate event for any attribute change.
Would like to know how would you design this scenario.
Thanks.
