Celery eventlet worker threads using too many database connections

Question

I have 2 celery workers which pool via eventlet, config is below:

celery multi start w1 w2 -A proj -l info --time-limit=600 -P eventlet -c 1000

When running more than 100 tasks at a time, I get hit by the error:

OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections

I'm running on PostgreSQL with max. connections set at the default of 100.

From what I read online, I thought worker threads in the pools would share the same DB connection. However, mine seem to try and create one connection per thread, which is why the error occurs.

Any ideas?

Thanks!

have you monkeypatched for psycopg? http://eventlet.net/doc/patching.html#monkeypatching-the-standard-library — Anentropic, Jun 14 '17 at 15:43

score 1 · Accepted Answer · answered Jun 15 '17 at 05:37

1

Django has (or had?) idle DB connection reuse to avoid overhead of creating new connection for each request. Idle reuse is not relevant in this scenario.

Django never had limiting DB connection pool. (please correct if wrong)

Consider overall design:

how many tasks do you need to execute concurrently? (real numbers are often not nice powers of 10)
how many simultaneous connections from this application can your database sustain?
do you need to place artificial bottlenecks (pools) or do you need to increase limits and use available hardware?

Consider using external [Postgresql connection pool] (google terms in square braces) or include one somewhere in your application.

answered Jun 15 '17 at 05:37

temoto

5,394
3
34
50

Thanks @temoto, I did a quick search and **will be trying out transaction pooling with PgBouncer** for a start. As my application is in Beta, I **do not yet have a very good estimate on the quantity of concurrent tasks and simultaneous connections** but I do know that they come in large batches instead of small steady flows. Couple quick questions: **1. By artifical bottlenecks (pools)**, are you referring to the PostgreSQL connection pool? **2. Increasing limits** would be to increase max connections in PostgreSQL also update related settings to support such an increase? – Leb Jun 15 '17 at 07:40
Pools refer to any pools, including pgbouncer or 20 lines code pool in your application. Increasing limits would be finding what's bottleneck limit right now and increase it, including postgresql config, application, OS kernel and any else. – temoto Jun 16 '17 at 08:23

Celery eventlet worker threads using too many database connections

1 Answers1