1

I am developing a solution that queries a SOAP web service for certain transactions. Once retrieved, these transactions are meant to be saved in a database after which a callback url is invoked to send some data to another server. How would you best architect a solution to this problem. My point of confusion is whether gen_server or gen_fsm should be used and if so which component of the solution goes where i.e. if gen_server what task goes to the server which task goes to the client.

Waliaula Makokha
  • 815
  • 1
  • 11
  • 21

1 Answers1

3

Think about what tasks can happen in parallel in your system. Can the SOAP queries be done in parallel, and does it make sense? Can the transactions be written to the database while SOAP is being queried again? Is the data to another server supposed to be sent after the transactions are written to the database, or can it be done at the same time?

Once you know these answers, you can build your pipeline. One example would be:

  • One process that regularly queries the SOAP service and calls a function with each batch of transactions.
  • In this function, two processes are started; one that writes the transactions to the database and one that sends data to the server. These processes are independent of each other.

This is just one example, and depending on your requirements, you might have to structure things another way.

In general, these processes can all be gen_servers as none of them have any clear states. In fact, the two worker processes doesn't have to be gen_servers at all, since they just do one task and then die. Using a gen_server in that case would be overkill.

Adam Lindberg
  • 16,447
  • 6
  • 65
  • 85
  • Thanks! Sorry for the vagueness on my part but your answer has helped clarify some of my thoughts. Just to point out that the data is sent via the url after it has been saved to the database. I was thinking of using a gen_fsm for querying the web service (alternate between two states i.e. fetching, fetched. Then sending the data to a gen_server which dynamically starts two gen_fsms one for saving the data and another for sending the data via url. I know for sure that the gen_fsms might be overkill but the component that queries the web service is some sort of a client to the gen_server. – Waliaula Makokha Aug 26 '15 at 20:44
  • 1
    I think a `gen_fsm` would be overkill in that case, because in the 'fetching' state you'd just call the SOAP library in that process so it would be busy anyway. Since it only sends data to other processes, there's no need for anyone else to know it's "state". Look at it more as a producer, than a state machine. – Adam Lindberg Aug 27 '15 at 07:49
  • 1
    And if you want to first write to the database, then send the data to the other server, I'd do that in one process, and not two (if it can be done sequentially, that is). That gives you a much simpler and predictable system. I'd even go as far as not making this a `gen_server` at all but just as simple worker monitored by a `simple_one_for_one` supervisor. – Adam Lindberg Aug 27 '15 at 07:50
  • Just for clarification you mean I have one worker that queries the web service managed by a one_for_one supervisor and another simple worker for saving to database and invoking the url managed by a simple_one_for_one supervisor (since the workers are spawned dynamically for each transaction retrieved). Kindly advise. – Waliaula Makokha Aug 27 '15 at 20:58
  • Yes, that would be a good way to do it. When thinking about supervisor structures it's also important to think about what needs to be supervised, and how it depends on other parts. In this case, your worker don't really depend on the query process (because they already have the data they need) and the query process doesn't really depend on the started workers (it just starts new ones), so they don't need to be under the same supervisor. They should probably be under the same supervision tree though. – Adam Lindberg Aug 28 '15 at 18:40