4

I am looking to implement a web server that will receive streaming data for all US Equities (approximately 10,000). I was wondering if there are any particular infrastructures that are best suited for this. I was planning to use a Gunicorn server with Gevent.

One bottleneck would be to save this data to a database. Would it be better to save data first to a Redis stream and send it by batch to the database to avoid overloading the CPU? Saving directly every WebSocket message for every stock would require too much resources.

Pier-Olivier Marquis
  • 419
  • 5
  • 14
  • It is an interesting question. Could you specific the total size of records and size of single record e.g. KB, MB, GB – The blank Oct 26 '21 at 12:54
  • @Theblank For the most active stocks, it can easily be a few million messages per day. Each of them would amount to perhaps 100 bytes or so. The issue, IMO, is really about the frequency. It can easily be several thousand messages per second. If I have a multithreaded server, the number of database connections would quickly become an issue. – Pier-Olivier Marquis Oct 26 '21 at 13:08
  • 1
    In my case, I use spring websocket to handle the messages of clients. I know that each client is held by a thread of server. If no messages sent, they are in waiting state. Processing power of each message is similar to processing power of traditional HTTP Request, but it still very hard to estimate hardware requirement. Not official answer, only personal experience – The blank Oct 26 '21 at 13:36
  • @Theblank I guess I will have to do some testing to find out. Thanks. – Pier-Olivier Marquis Oct 26 '21 at 14:00

0 Answers0