Django channels "ERROR Y of N channels over capacity in group subscriptions"

Question

I'm doing load testing with my Django app providing GraphQL Subscriptions using Django channels and a redis Channels layer (django, graphene-django, channels, graphene-subscriptions, channels-redis). As ASGI server I'm using daphne right now. I use nginx as proxy. The periodicity with which the backend publishes messages via GraphQL Subscriptions depends on the periodicity of messages the backend receives via MQTT. I'm increasing the periodicity with which an external data provider publishes messages to the MQTT broker, means the periodicity with which the backend has to process these messages and publish messages via GraphQL Subscriptions. I'm facing the following error:

2020-03-11 08:33:58,464 ERROR    2 of 12 channels over capacity in group subscriptions

It seems like this issue is caused by channels_redis. Can I scale the infrastructure to workaround this issue?

from that issue it looks like you might be able to adjust the default number in your configuration. Have you attempted to do this? — Matthaus Woolard, Mar 11 '20 at 22:13
Probably I have to adjust and optimize `capacity` and `expire` in the `CHANNELS_LAYERS` configuration. Despite of that configuration option it's likely that the backend does not get and process queued messages fast enough. Means I have to improve the backend's performance. — thinwybk, Mar 12 '20 at 09:19

score 14 · Accepted Answer · answered Jun 30 '20 at 12:23

The default capacity is 100 messages and the default message expiration time is 60 seconds. So if the the channel is never read within these capacity / time constraints, it will fill up.

One reason why a channel might fill up is when the connection is never properly closed. In this case the channel will remain in the group and eventually fill up.

One way to mitigate this is to have enough capacity and a short timeout. You can alter the configuration in the Django settings in the following way:

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": REDIS_URL,  # or where your redis server lives
            "capacity": 1500,  # default 100
            "expiry": 10,  # default 60
        }
     }
}

Your answer guided me to the actual root cause. I've a event driven system where the data has to be saved into a database before the data in the cache is "cleared". I was using a development SQLite database instead of a production database. Migration to a production database with better performance resolved the issue. — thinwybk, Jul 13 '20 at 06:59

Django channels "ERROR Y of N channels over capacity in group subscriptions"

1 Answers1