Sorry for the novel...
I'm working on a Nodejs project where I need to decrypt millions of envelopes in multiple files. Any APIs of my application have to run on localhost.
The main API handles client requests to decrypt a batch of files. Each file contains thousands to millions of envelopes that need to be decrypted. Each file is considered a job and these jobs are queued up by the Main API and then run concurrently by forking a new process for each job. (I only allow 5 concurrent jobs/forks at one time) In each process, a script runs that goes through and decrypts the file.
This runs relatively quickly but instead of doing the decryption in the code of each process/script forked by the Main API, I want to dish this work out to another API (call it Decrypt API) that basically takes the envelope in the request and sends back the decrypted result in the response.
So I created this api and then used 'forky' to cluster it. Then from my processes, instead of doing the decryption in those, I makes multiple parallel requests to the Decrypt API and once I get the responses back just place the decrypted results in a file.
At first my problem was that I made requests right as I got each envelope without waiting for a request to return before sending the next one. I would basically send "parallel" requests if you will, and then just handle the vote in the callback of each request. This led to what I think is too many outstanding reqs at one time because I was getting an ECONNRESET error. Some requests were dropped. So my solution was to have a maximum of x outstanding reqs(I used 10) at any one time to avoid too many concurrent reqs. This seemed ok but then I realized since I was forking 5 processes from the MainAPI, and although each one had this new 'outstanding reqs' limiting code, since they were running concurrently I was still running into the problem of too many reqs at once to the Decrypt API. Also, this method of using two different microservices/APIs is slower than just having the MainAPI's forked processses just do the decryption. In the Decrypt API I'm also using the node 'crypto' library and some of those functions that I use are synchronous so I suspect that with high traffic that's a problem, but I can't avoid those sync methods.
So finally, my question is, what can I do to increase the speed of the Decrypt API with high traffic like I described, and what can I do to avoid these dropped requests?
Forgive me if I sound like a noob, but since these APIs are all running on the same machine and localhost, could this be why this method is slower than just doing decryption in each process?
Thanks!
