Django 2.0 - test using fixtures from dumpdata gives utf8mb4 encoding errors

Question

I am using utf8mb4 encoding for my mySQL database. All my Django apps work correctly with it even with Cyrillic characters.

But when I run ./manage.py test <module> and try to load a fixture dumped from the same database, I get errors regarding the encoding. One of the errors is:

Could not load advert.Region(pk=1): (1366, "Incorrect string value: '\\xD0\\x9F\\xD0\\xBB\\xD0\\xBE...' for column 'region_name' at row 1")

Then I set DEFAULT_CHARSET in the Django settings and the error changed to LookupError: Problem installing fixture '<path>/test_db_one.xml': unknown encoding: utf8mb4

As you can see I have specified the format of the fixture to be XML as it has better handling of the encoding.

My setting module has the following properties:

DEFAULT_CHARSET = 'utf8mb4'
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'autoplusplus_dev',
        'USER': 'autoplusplus_django',
        'PASSWORD': '1029',
        'HOST': 'localhost',
        'PORT': '3306',
        'STORAGE_ENGINE': 'INNODB',
        'OPTIONS': {'charset': 'utf8mb4'},
        'TEST_CHARSET': 'utf8mb4',
    }
}

I have mysqlclient 1.3.12 installed

score 1 · Accepted Answer · answered Jan 02 '18 at 14:40

1

Turned out that changing the encoding of the file which Django generates itself to UTF-8 fixes the issue.

answered Jan 02 '18 at 14:40

Teddy Markov

266
3
15

score 0 · Answer 2 · answered Dec 27 '17 at 17:27

0

You are expecting Пло? If so, that is reasonable UTF-8 encoding.

"unknown encoding: utf8mb4" -- You have an antique version of Django and/or MySQL? What versions are you running?

What does the table definition look like? Maybe the columns are latin1 instead of utf8 or utf8mb4.

Meanwhile, use MySQL's CHARACTER SET utf8; this may get you going.

answered Dec 27 '17 at 17:27

Rick James

135,179
13
127
222

Thanks for the help. Turned out I had to change the encoding of the xml file itself and this fixed the issue. – Teddy Markov Jan 02 '18 at 14:38
Hmmm... I thought XML had to be UTF-8. – Rick James Jan 02 '18 at 16:53

Django 2.0 - test using fixtures from dumpdata gives utf8mb4 encoding errors

2 Answers2