There is a download page for about 5000 sql queries against a database (Postrgres). The team decided to apply the following algorithm to avoid 502 errors when multiple users load the page at the same time (using Redis):
- Create a task for an additional worker
- Connect via WebSocket to the page. (or use an AJAX request)
- While the request is running, the user sees the download window.
- When the worker completes the calculation, it will pass the result to the consumers
The project uses django_rq. Are there ready-made solutions that perform this algorithm? Like a decorator?
Also, if you have suggestions for optimizing such a number of requests, please suggest them.
Used standard Django optimization tools to reduce the number of requests.