0

I deployed my Django project that uses Apache and MySQL. all apps features work fine, but in an application's views.py, when I try to access it, it raises this error.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9735: ordinal not in range(128)

it works fine in localhost , my query contains some arabic character

I searched a lot but none of them works for me. The views.py of the app has many queries (for calculating), annotate and aggregate also Case(When())?

this is my error.log

[Fri Aug 21 14:30:46.907352 2020] [wsgi:error] [pid 20969:tid 140683116439296] [remote 95.159.84.254:12151]   File "/var/www/projectname/venv/lib/python3.6/site-packages/django/views/debug.py", line 94, in technical_500_response
[Fri Aug 21 14:30:46.907356 2020] [wsgi:error] [pid 20969:tid 140683116439296] [remote 95.159.84.254:12151]     html = reporter.get_traceback_html()
[Fri Aug 21 14:30:46.907362 2020] [wsgi:error] [pid 20969:tid 140683116439296] [remote 95.159.84.254:12151]   File "/var/www/projectname/venv/lib/python3.6/site-packages/django/views/debug.py", line 332, in get_traceback_html
[Fri Aug 21 14:30:46.907379 2020] [wsgi:error] [pid 20969:tid 140683116439296] [remote 95.159.84.254:12151]     t = DEBUG_ENGINE.from_string(fh.read())
[Fri Aug 21 14:30:46.907387 2020] [wsgi:error] [pid 20969:tid 140683116439296] [remote 95.159.84.254:12151]   File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
[Fri Aug 21 14:30:46.907391 2020] [wsgi:error] [pid 20969:tid 140683116439296] [remote 95.159.84.254:12151]     return codecs.ascii_decode(input, self.errors)[0]
[Fri Aug 21 14:30:46.907402 2020] [wsgi:error] [pid 20969:tid 140683116439296] [remote 95.159.84.254:12151] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9735: ordinal not in range(128)
[Fri Aug 21 14:30:50.086155 2020] [wsgi:error] [pid 20969:tid 140683116439296] [remote 95.159.84.254:12181] Not Found: /favicon.ico
[Fri Aug 21 11:36:30.228905 2020] [mpm_event:notice] [pid 20966:tid 140683347807168] AH00491: caught SIGTERM, shutting down
[Fri Aug 21 11:36:30.330450 2020] [mpm_event:notice] [pid 21882:tid 140691412487104] AH00489: Apache/2.4.29 (Ubuntu) mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Fri Aug 21 11:36:30.330585 2020] [core:notice] [pid 21882:tid 140691412487104] AH00094: Command line: '/usr/sbin/apache2'
[Fri Aug 21 12:15:09.336685 2020] [mpm_event:notice] [pid 21882:tid 140691412487104] AH00491: caught SIGTERM, shutting down
[Fri Aug 21 12:15:09.451585 2020] [mpm_event:notice] [pid 22113:tid 139983902116800] AH00489: Apache/2.4.29 (Ubuntu) mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Fri Aug 21 12:15:09.451973 2020] [core:notice] [pid 22113:tid 139983902116800] AH00094: Command line: '/usr/sbin/apache2'
[Fri Aug 21 12:51:39.616795 2020] [mpm_event:notice] [pid 22113:tid 139983902116800] AH00491: caught SIGTERM, shutting down
[Fri Aug 21 12:51:39.714991 2020] [mpm_event:notice] [pid 22639:tid 140292123478976] AH00489: Apache/2.4.29 (Ubuntu) mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Fri Aug 21 12:51:39.715191 2020] [core:notice] [pid 22639:tid 140292123478976] AH00094: Command line: '/usr/sbin/apache2'
art_cs
  • 683
  • 1
  • 8
  • 17

1 Answers1

1

If you are converting to or from UTF-8, it does have a specific format that must be followed.

If the data that has been submitted is not submitted in UTF-8 format, then it will not correctly decode as UTF-8

That is, if the international characters (non 7-bit characters) have been encoded using a different text-encoding, then

  1. its likely they will not decode as UTF-8, giving the error you are seeing
  2. the decoded data will come out completely wrong - its like translating English to French, but then translating [the French] back to English by treating it as German - the result will be errors & nonsense.

For any modern system, especially one that you want to be fully multi-lingual, you should ensure all data is always submitted as UTF-8

libidn2 has the function u8_check that will do a UTF-8 validation on any UTF-8 encoded data.

Also, NOTE - "utf8" in MySQL is NOT correct UTF-8, its an early implementation that is not fully compliant.

You should use utf8mb4 instead - using the older one could also cause similar problems to what you have - https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434

James Stevens
  • 374
  • 2
  • 8
  • CREATE DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; i used it make my db , still i dont entered any data , zero database , i have used some arabic character , in my query – art_cs Aug 21 '20 at 15:32
  • Looks like the file being read `fh.read()` has been opened as `ascii` not `utf8` - try looking at this - https://stackoverflow.com/questions/18649512/unicodedecodeerror-ascii-codec-cant-decode-byte-0xe2-in-position-13-ordinal – James Stevens Aug 22 '20 at 17:11