paramiko-1.16.0 readlines() decode error

Question

Doing a fairly standard paramiko implementation returns decode error on readlines() for oddly formed text from remote command. I can not change the command output. How can I write the code to properly decode. See decode error text at bottom:

code snip being used:

connect = paramiko.SSHClient()
connect.connect(self.name,
                username = self.ruser,
                password = password,
                key_filename = idkey,
                timeout = 15,
               )
stdin, stdout, stderr = connect.exec_command(cmd)
retval=stdout.channel.recv_exit_status()
stdin.flush()
stdout.flush()
stderr.flush()
out = stdout.readlines()
sys.exit()

ERROR TEXT:

File "../pylib/hosts/host.py", line 128, in cmd
out = stdout.readlines()
File "/usr/local/lib/python3.4/site-packages/paramiko/file.py", line 285, in readlines
line = self.readline()
File "/usr/local/lib/python3.4/site-packages/paramiko/file.py", line 270, in readline
return line if self._flags & self.FLAG_BINARY else u(line)
File "/usr/local/lib/python3.4/site-packages/paramiko/py3compat.py", line 148,
in u return s.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 80: invalid continuation byte

score 2 · Answer 1 · answered Jan 01 '16 at 22:29

Well, I modified the paramiko.py3compat "u" method for python 3. (Did not modify python 2 version) If the utf8 decode fails, try ISO-8859-1. Standard try: except: wrapper.

Suspect this issue only occurs with Python 3. Would have been nice to overide the py3compay.pt u() method instead of modifying the paramiko library file, but needed to move on from this issue.

Here is the new u() method I patched into py3compat.py:

def u(s, encoding='utf8'):
    """cast bytes or unicode to unicode"""
    if isinstance(s, bytes):
        try:
            return s.decode(encoding)
        except UnicodeDecodeError:
            return s.decode('ISO-8859-1')
    elif isinstance(s, str):
        return s
    else:
        raise TypeError("Expected unicode or bytes, got %r" % s)

score 0 · Answer 2 · answered Sep 01 '23 at 05:47

Based on Fragtzack, here is a more flexible way.

def u(s, encoding="utf8"):
    """cast bytes or unicode to unicode"""
    if isinstance(s, bytes):
        #return s.decode(encoding)
        try:
            return s.decode(encoding)
        except UnicodeDecodeError:
            return decoding(s)
    elif isinstance(s, str):
        return s
    else:
        raise TypeError("Expected unicode or bytes, got {!r}".format(s))

def decoding(s, encodings=[ "utf-8", "cp936"]):
    """cast bytes to unicode by encodings"""
    for encoding in encodings:
        try:
            return s.decode(encoding)
        except UnicodeDecodeError as lastError:
            continue
    else:
        raise lastError

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). — Diego Borba, Sep 01 '23 at 13:25

paramiko-1.16.0 readlines() decode error

2 Answers2