Set filesystem encoding for python 3 on the Intel Edison

Question

Steps to reproduce:

1. Create a file test.txt with content This is 中文 (i.e., UTF-8 encoded non-ASCII text).
2. Custom-compile python 3.5.2 on the Intel Edison.

3. Launch the custom-compiled python3 interpreter and issue the following piece of code:

   with open('test.txt', 'r') as fh:
       fh.readlines()
   

Actual behavior:

A UnicodeDecodeError exception is thrown. The file is opened as 'ASCII' instead of 'UTF-8' by default:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 8: ordinal not in range(128)

On a "regular" Linux system this problem is easily solved by setting a proper locale, see e.g. this post or that post. On the Intel Edison, however, I cannot set the LC_CTYPE since the default Yocto Linux distribution is missing locales (see e.g. this page).

I also tried to use a couple of other hacks like

import sys; sys.getfilesystemencoding = lambda: 'UTF-8'
import locale; locale.getpreferredencoding = lambda: 'utf-8'

And I tried setting the PYTHONIOENCODING=utf8 environment variable before starting the python interpreter.

However, none of this works. The only workaround is to specify the encoding explicitly as a command line parameter to the open command. This works for the above snippet, but it won't set the system-wide default for all packages I am using (that will implicitly open files as ASCII and may or may not offer me a way to override that default behavior).

What is the proper way to set the python interpreter default filesystem encoding? (Of course without installing unneeded system-wide locales.)

Answer 1

You can set the LC_ALL environment variable to alter the default:

$ python3 -c 'import locale; print(locale.getpreferredencoding())'
UTF-8
$ LC_ALL='.ASCII' python3 -c 'import locale; print(locale.getpreferredencoding())'
US-ASCII

I tested this both on OS X and CentOS 7.

As for your other attempts, here is why they don't work:

• sys.getfilesystemencoding() applies to filenames only (e.g. os.listdir() and friends).
• The io module doesn't actually use the locale.getpreferrredencoding() function, so altering the function on the module won't have an effect. A lightweight _bootlocale.py bootstrap module is used instead. More on that below.
• PYTHONIOENCODING only applies to sys.stdin, sys.stdout and sys.stdstderr

If setting environment variables ultimately fails, you can still patch the _bootlocale module:

import _bootlocale

old = _bootlocale.getpreferredencoding  # so you can restore
_bootlocale.getpreferredencoding = lambda *args: 'UTF-8'

This works for me (again on OS X and CentOS 7, tested with 3.6):

>>> import _bootlocale
>>> open('/tmp/test.txt', 'w').encoding  # showing pre-patch setting
'UTF-8'
>>> old = _bootlocale.getpreferredencoding
>>> _bootlocale.getpreferredencoding = lambda *a: 'ASCII'
>>> open('/tmp/test.txt', 'w').encoding  # gimped hook
'ASCII'