GROUP_CONCAT equivalent in Django

Question

Say I have the following table called fruits:

id | type   | name
-----------------
 0 | apple  | fuji
 1 | apple  | mac
 2 | orange | navel

My goal is to ultimately come up with a count of the different types and a comma-delimited list of the names:

apple, 2, "fuji,mac"
orange, 1, "navel"

This can be easily done with GROUP_CONCAT in MySQL but I'm having trouble with the Django equivalent. This is what I have so far but I am missing the GROUP_CONCAT stuff:

query_set = Fruits.objects.values('type').annotate(count=Count('type')).order_by('-count')

I would like to avoid using raw SQL queries if possible.

Any help would be greatly appreciated!

Thanks! =)

score 57 · Answer 1 · edited Sep 19 '21 at 23:25

57

You can create your own Aggregate Function (doc)

from django.db.models import Aggregate

class Concat(Aggregate):
    function = 'GROUP_CONCAT'
    template = '%(function)s(%(distinct)s%(expressions)s)'

    def __init__(self, expression, distinct=False, **extra):
        super(Concat, self).__init__(
            expression,
            distinct='DISTINCT ' if distinct else '',
            output_field=CharField(),
            **extra)

and use it simply as:

query_set = Fruits.objects.values('type').annotate(count=Count('type'),
                       name = Concat('name')).order_by('-count')

I am using django 1.8 and mysql 4.0.3

edited Sep 19 '21 at 23:25

Ronan Boiteau

9,608
6
34
56

answered Jul 10 '15 at 09:43

Shashank Singla

1,797
17
13

1

NOTICE that Django (>=1.8) provides [`Database functions`](http://stackoverflow.com/a/40478702/2714931) – WeizhongTu Jan 24 '17 at 06:05
2

For the sake of completeness, there is also: `django.contrib.postgres.aggregates.StringAgg` in case you want the same in Postgres – Lorenzo Peña Jun 28 '19 at 20:33
3

This function is also present in `django-mysql`: https://django-mysql.readthedocs.io/en/latest/aggregates.html?highlight=GroupConcat#django_mysql.models.GroupConcat – Alexander Sep 23 '19 at 09:17
4

For django 2.2 I needed to add `allow_distinct = True` to Concat class – Ali Dabour Nov 16 '19 at 12:05
@LorenzoPeña That is the only thing which worked amongst the jungle of Group Concat articles and answers. You should add that as an answer `StringAgg(fieldname, delimiter)` – Akshay Hazari Dec 31 '20 at 07:50
1

A word of caution: MySQL will truncate GROUP_CONCAT result to 1024 characters, by default - I've spent some time figuring out why I was getting non-existent ids. You can set group_concat_max_len, example is here: https://stackoverflow.com/questions/36475140/include-multiple-statements-in-djangos-raw-queries – Soid Aug 24 '21 at 20:14
Note: if someone faces not defined issue of `CharField` in line `output_field=CharField(),` then you may just replace `from django.db.models import Aggregate` to `from django.db.models import Aggregate, CharField` – miltonbhowmick Oct 19 '22 at 11:15

score 20 · Answer 2 · edited Sep 19 '21 at 23:24

NOTICE that Django (>=1.8) provides Database functions support. https://docs.djangoproject.com/en/dev/ref/models/database-functions/#concat

Here is an enhanced version of Shashank Singla

from django.db.models import Aggregate, CharField

class GroupConcat(Aggregate):
    function = 'GROUP_CONCAT'
    template = '%(function)s(%(distinct)s%(expressions)s%(ordering)s%(separator)s)'

    def __init__(self, expression, distinct=False, ordering=None, separator=',', **extra):
        super(GroupConcat, self).__init__(
            expression,
            distinct='DISTINCT ' if distinct else '',
            ordering=' ORDER BY %s' % ordering if ordering is not None else '',
            separator=' SEPARATOR "%s"' % separator,
            output_field=CharField(),
            **extra
        )

Usage:

LogModel.objects.values('level', 'info').annotate(
    count=Count(1), time=GroupConcat('time', ordering='time DESC', separator=' | ')
).order_by('-time', '-count')

It doesn;t work for me. Another fine example: https://gist.github.com/ludoo/ca6ed07e5c8017272701 — Ilia w495 Nikitin, Apr 18 '17 at 19:42
@Iliaw495Nikitin this works well in my project using Django 1.10.x — WeizhongTu, Apr 19 '17 at 02:42

Adam Johnson · Answer 3 · 2020-01-06T17:31:02.670

9

Use GroupConcat from the Django-MySQL package ( https://django-mysql.readthedocs.org/en/latest/aggregates.html#django_mysql.models.GroupConcat ) which I maintain. With it you can do it simply like:

>>> from django_mysql.models import GroupConcat
>>> Fruits.objects.annotate(
...     count=Count('type'),
...     name_list=GroupConcat('name'),
... ).order_by('-count').values('type', 'count', 'name_list')
[{'type': 'apple', 'count': 2, 'name_list': 'fuji,mac'},
 {'type': 'orange', 'count': 1, 'name_list': 'navel'}]

edited Jan 06 '20 at 17:31

answered Mar 05 '16 at 16:22

Adam Johnson

471
1
5
11

I know this is an old answer but I am somehow confused. May be am missing something. So how does the GroupConcat know that it will concatenate values of field 'name' ? As I don't see you indicating the field 'name' anywhere in the query, and yet we are concatenating values in field 'name' – manpikin Jan 05 '20 at 16:34
Great! Your answer was helpful. I will give you a vote. :) @adam-chainz – manpikin Jan 06 '20 at 17:59

score 4 · Answer 4 · answered Jun 22 '18 at 16:22

4

If you are using PostgreSQL, you can use ArrayAgg to aggregate all of the values into an array.

https://www.postgresql.org/docs/9.5/static/functions-aggregate.html

answered Jun 22 '18 at 16:22

Jin

6,055
2
39
72

daseme · Answer 5 · 2012-05-04T16:20:11.070

3

If you don't mind doing this in your template the Django template tag regroup accomplishes this

edited May 04 '12 at 16:20

answered May 04 '12 at 16:05

daseme

31
1
2

score 3 · Answer 6 · edited Jun 22 '20 at 20:55

3

As of Django 1.8 you can use Func() expressions.

query_set = Fruits.objects.values('type').annotate(
    count=Count('type'),
    name=Func(F('name'), 'GROUP_BY')
).order_by('-count')

edited Jun 22 '20 at 20:55

Diego Allen

4,623
5
30
33

answered Dec 01 '15 at 17:08

Joel Davis

47
2

score 2 · Accepted Answer · answered Apr 26 '12 at 20:33

2

The Django ORM does not support this; if you don't want to use raw SQL then you'll need to group and join.

answered Apr 26 '12 at 20:33

Ignacio Vazquez-Abrams

776,304
153
1,341
1,358

2

A colleague of mine maintains an open source project that exposes mysql specific functions like GROUP_CONCAT in django. Take a look https://github.com/adamchainz/django-mysql/ – Michael Aquilina Jul 23 '15 at 13:36
Just note that for Postgres we have django.contrib.postgres.aggregates.StringAgg. – mirek Dec 18 '20 at 20:43

frnhr · Answer 8 · 2016-09-27T15:27:06.863

Not supported by Django ORM, but you can build your own aggregator.

It's actually pretty straightforward, here is a link to a how-to that does just that, with GROUP_CONCAT for SQLite: http://harkablog.com/inside-the-django-orm-aggregates.html

Note however, that it might be necessary to handle different SQL dialects separately. For example, the SQLite docs say about group_concat:

The order of the concatenated elements is arbitrary

While MySQL allows you to specify the order.

I guess that may be a reason why GROUP_CONCAT it's not implemented in Django at the moment.

score 1 · Answer 9 · answered Mar 18 '19 at 07:47

To complete the answer of @WeizhongTu, Notice that you can not use the keyword SEPARATOR with SQLITE. In cases where you are using MySQL and SQLite for your tests, you can write :

class GroupConcat(Aggregate):
    function = 'GROUP_CONCAT'
    separator = ','

    def __init__(self, expression, distinct=False, ordering=None, **extra):
        super(GroupConcat, self).__init__(expression,
                                          distinct='DISTINCT ' if distinct else '',
                                          ordering=' ORDER BY %s' % ordering if ordering is not None else '',
                                          output_field=CharField(),
                                          **extra)

    def as_mysql(self, compiler, connection, separator=separator):
        return super().as_sql(compiler,
                              connection,
                              template='%(function)s(%(distinct)s%(expressions)s%(ordering)s%(separator)s)',
                              separator=' SEPARATOR \'%s\'' % separator)

    def as_sql(self, compiler, connection, **extra):
        return super().as_sql(compiler,
                              connection,
                              template='%(function)s(%(distinct)s%(expressions)s%(ordering)s)',
                              **extra)

score 1 · Answer 10 · answered Aug 24 '21 at 20:22

I just wanted to say a word of caution if you go with any of the proposed solutions for MySQL: by default, MySQL will truncate GROUP_CONCAT result to 1024 characters. I've spent some time figuring out why I was getting non-existent ids (they were truncated existent ids).

You can avoid the limitation by setting group_concat_max_len in Django settings. An example is here: Include multiple statements in Django's raw queries

score 0 · Answer 11 · edited Jan 11 '22 at 03:04

0

this is the best way of working in Django ORM

f1 = Fruits.objects.values('type').annotate(count = Count('type'),namelist= GroupConcat('namelist')).distinct()

edited Jan 11 '22 at 03:04

Saman Salehi

1,004
1
12
19

answered Jan 10 '22 at 13:58

Narender Parikapalli

19
4

GROUP_CONCAT equivalent in Django

11 Answers11

Linked