12

Using Python 3.4.3 on Windows.

My script runs a little java program in console, and should get the ouput:

import subprocess
p1 = subprocess.Popen([ ... ], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
out, err = p1.communicate(str.encode("utf-8"))

This leads to a normal

'UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 135: character maps to < undefined>'.

Now I want to ignore errors:

out, err = p1.communicate(str.encode(encoding="utf-8", errors="ignore"))

This leads to a more interesting error I found no help for using google:

TypeError: descriptor 'encode' of 'str' object needs an argument

So it seems that python does not even know anymore what the arguments for str.encode(...) are. The same also applies when you leave out the errors part.

user136036
  • 11,228
  • 6
  • 46
  • 46
  • https://stackoverflow.com/a/33291200/13689597 hey bro I met this problem too, thanks for your detail answer. – GoogTech Sep 23 '20 at 07:38

3 Answers3

25

universal_newlines=True enables text mode. Combined with stdout=PIPE, it forces decoding of the child process' output using locale.getpreferredencoding(False) that is not utf-8 on Windows. That is why you see UnicodeDecodeError.

To read the subprocess' output using utf-8 encoding, drop universal_newlines=True:

#!/usr/bin/env python3
from subprocess import Popen, PIPE

with Popen(r'C:\path\to\program.exe "arg 1" "arg 2"',
           stdout=PIPE, stderr=PIPE) as p:
    output, errors = p.communicate()
lines = output.decode('utf-8').splitlines()

str.encode("utf-8") is equivalent to "utf-8".encode(). There is no point to pass it to .communicate() unless you set stdin=PIPE and the child process expects b'utf-8' bytestring as an input.

str.encode(encoding="utf-8", errors="ignore) has the form klass.method(**kwargs). .encode() method expects self (a string object) that is why you see TypeError.

>>> str.encode("abc", encoding="utf-8", errors="ignore") #XXX don't do it
b'abc'
>>> "abc".encode(encoding="utf-8", errors="ignore")
b'abc'

Do not use klass.method(obj) instead of obj.method() without a good reason.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
2

You are not supposed to call .encode() on the class itself. What you probably want to do is something like

p1.communicate("FOOBAR".encode("utf-8"))

The error message you're getting means that the encode() function has nothing to encode, since you called it on the class, rather than on an instance (that would then be passed as the self parameter to encode()).

Colin Emonds
  • 765
  • 4
  • 18
  • The problem is: if you use communicate( str.encode("utf-8") ) this works fine (as can be seen on several stackoverflow examples), except some unicode errors. But when you add the error argument or use encoding="utf-8" it breaks. Otherwise it works fine. – user136036 Oct 22 '15 at 15:58
  • 1
    That's because what `std.encode("utf-8")` really does is the same as `"utf-8".encode()`, which returns... "utf-8" :) So you're actually sending the text "utf-8" to your app - I don't think that's what you wanted to do. – viraptor Oct 22 '15 at 21:53
0

If using Popen to run another Python script, then see my answer.

The short answer is set the enviornment variable PYTHONIOENCODING, and set encoding='utf-8' in Popen.

Ruhollah Majdoddin
  • 29
  • 4