Data pull and push

Question

I am writing a .NET window service which schedules job in certain interval. I am using QUARTZ for scheduling. A job will do 2 tasks:

1. Pull data using some SDK from one data source.
2. Push retrieved data to some other data source using REST APIs.

Pulling data is time consuming and I will pull in chunks and push the same.

My question is mostly on design side on the component who sits between push and pull. I want to notify push component when something becomes available for push.

Please let me know opinions on job queue, observer?

Answer 1

I would you use an hexagonal approach. In details:

• the Quartz job would be an object that completely delegates the work to a dedicated use case object (e.g. NotifyExtractedData, probably you can think of a better name...) in order to avoid coupling between Quartz details and your business logic;
• NotifyExtractedData object would depend upon two interfaces that represent the ports of the hexagon: may be IDataGateway and INotifationGateway
• IDataGateway could have a method Chunk GetChunk(); The implementation of this interface could be an object that represent the hexagon adapter to the SDK: SDKDataGateway. Inside this object I would put all the logic of data extraction using the SDK. Chunk would be a DTO that contains the data chunk of your extraction;
• INotificationGateway interface could have a method void Notify(Chunk chunck); that receives the data chunk extracted by your data gateway. The implementation of the INotificationGateway interface could be an object RESTNotificationGateway that contains all the details about REST API you need to use to notify data.

This is only a raw idea of a possible solution. The names could be improved for sure. I hope it could help you.