Concatenate unicode and utf-8 Python 2.7

Question

>>> dir = u'\\\\nas\\cut\\'
>>> cutter = "seleção"
>>> ext = ".cf2"
>>> path = dir+cutter+ext

Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    path = dir+cutter+ext
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 4: ordinal not in range(128)

This is needed to use:

f = open(dir, 'r')

I don't know how I concatenate this well. The variable dir must be in unicode because I use configparser to parse te value from a .ini file and that comes with unicode encoding.

Martijn Pieters · Answer 1 · 2016-01-20T08:07:36.600

4

Decode your bytes to a unicode string, explicitly:

path = dir + cutter.decode('utf8') + ext.decode('utf8')

Note that you should really use the os.path.join() function to build paths:

path = os.path.join(dir, cutter.decode('utf8') + ext.decode('utf8'))

This assumes you know your terminal or console is configured for UTF-8; it is better to use sys.stdin.encoding here. For data sourced from elsewhere, determine the codec for that source first.

edited Jan 20 '16 at 08:07

answered Jan 19 '16 at 09:42

Martijn Pieters

1,048,767
296
4,058
3,343

What if his terminal isn't UTF-8? :) – Mark Tolonen Jan 20 '16 at 03:12
@MarkTolonen good point; I did make an assumption there based on the tags. Using `sys.stdin.encoding` would be the better choice here. – Martijn Pieters Jan 20 '16 at 06:59

Mark Tolonen · Answer 2 · 2016-01-20T03:23:23.777

If your filename and extension are constants like your example, just use Unicode strings for everything:

>>> dir = u'\\\\nas\\cut\\'
>>> cutter = u"seleção"
>>> ext = u".cf2"
>>> path = dir+cutter+ext

If not constants and they are byte strings, .decode() them with an appropriate encoding. What that encoding is will be OS-dependent.

Note that some APIs like os.listdir() and glob.glob() can take a Unicode argument and return Unicode strings.

Concatenate unicode and utf-8 Python 2.7

2 Answers2