I am running an asynchronous Django server with django-postgrespool2 as its Database Engine on my local machine. No matter how I configure the pooling parameters, it creates only a certain amount of pool connections. This number seems to be directly proportional to the number of Uvicorn Workers I run for my application.
The following parameters are configured in the settings.py
DATABASE_POOL_CLASS = 'sqlalchemy.pool.QueuePool'
DATABASE_POOL_ARGS = {"max_overflow": 500, "pool_size": 1000, "recycle": 300}
I am using the QueuePool class of SQLAlchemy for queueing as mentioned in the official documentation of Django-Postgrespool2. Also, I have significantly incremented the pool size to 1000 and max_overflow to 500. After concurrently hitting requests with a load testing tool, what I have observed is, that the number of database connections in the pool only increase with the number of Uvicorn workers I have running.
Even though the pool size has been set to 1000, the number of pool connections created, are limited. My inferences, based on the number of Uvicorn workers that have been run and the number of pool connections created as a result of this are stated as follows:
| Number of Uvicorn Workers | Pool Connections Created |
|---|---|
| 5 | 80 |
| 6 | 96 |
| 7 | 112 |
| 8 | 128 |
| 9 | 144 |
| 10 | 160 |
| 50 | 773 |
The pool_size and max_overflow remained constant during my experiments. Also, these results were based on 50000 requests hit with 1000 concurrent threads.
As it can be seen that the requests (which are constant throughout) generate only a certain number of pool connections with increasing workers. Earlier, it never went past 100 but that changed when I altered the max_connections in postgresql.conf to 1000 from the default(100). But I still can't find a way out to increase number of pool connections per worker.
The pool_size parameter only helps limit the pool connections if the requests are way beyond its capacity but the backend application doesn't entirely utilise the pool size allotted to it. Is there any way I could achieve this without having to spawn a lot of workers?
