Python reading until null character from Telnet

Question

I am telneting to my server, which answers to me with messages and at the end of each message is appended hex00 (null character) which cannot be read. I tried searching through and through, but can't seem to make it work, a simple example:

from telnetlib import Telnet
connection = Telnet('localhost', 5001)
connection.write('aa\n')
connection.read_eager()

This returns an output:

'Fail - Command aa not found.\n\r'

whereas there should be sth like:

'Fail - Command aa not found.\n\r\0'

Is there any way to get this end of string character? Can I get bytes as an output if the character is missed on purpose?

The 00 character is there:

tcpdump

score 3 · Accepted Answer · answered Sep 16 '15 at 18:43

I stumbled in this same problem when trying to get data from an RS232-TCP/IP Converter using telnet - the telnetlib would suppress every 0x00 from the message. As Fredrik Johansson well answered, it is the way telnetlib was implemented.

One solution would be to override the process_rawq() function from telnetlib's Telnet class that doesn't eat all the null characters:

import telnetlib
from telnetlib import IAC, DO, DONT, WILL, WONT, SE, NOOPT

def _process_rawq(self):
    """Alteração da implementação desta função necessária pois telnetlib suprime 0x00 e \021 dos dados lidos
    """
    buf = ['', '']
    try:
        while self.rawq:
            c = self.rawq_getchar()
            if not self.iacseq:
#                if c == theNULL:
#                    continue
#                if c == "\021":
#                    continue
                if c != IAC:
                    buf[self.sb] = buf[self.sb] + c
                    continue
                else:
                    self.iacseq += c
            elif len(self.iacseq) == 1:
                # 'IAC: IAC CMD [OPTION only for WILL/WONT/DO/DONT]'
                if c in (DO, DONT, WILL, WONT):
                    self.iacseq += c
                    continue

                self.iacseq = ''
                if c == IAC:
                    buf[self.sb] = buf[self.sb] + c
                else:
                    if c == SB: # SB ... SE start.
                        self.sb = 1
                        self.sbdataq = ''
                    elif c == SE:
                        self.sb = 0
                        self.sbdataq = self.sbdataq + buf[1]
                        buf[1] = ''
                    if self.option_callback:
                        # Callback is supposed to look into
                        # the sbdataq
                        self.option_callback(self.sock, c, NOOPT)
                    else:
                        # We can't offer automatic processing of
                        # suboptions. Alas, we should not get any
                        # unless we did a WILL/DO before.
                        self.msg('IAC %d not recognized' % ord(c))
            elif len(self.iacseq) == 2:
                cmd = self.iacseq[1]
                self.iacseq = ''
                opt = c
                if cmd in (DO, DONT):
                    self.msg('IAC %s %d',
                        cmd == DO and 'DO' or 'DONT', ord(opt))
                    if self.option_callback:
                        self.option_callback(self.sock, cmd, opt)
                    else:
                        self.sock.sendall(IAC + WONT + opt)
                elif cmd in (WILL, WONT):
                    self.msg('IAC %s %d',
                        cmd == WILL and 'WILL' or 'WONT', ord(opt))
                    if self.option_callback:
                        self.option_callback(self.sock, cmd, opt)
                    else:
                        self.sock.sendall(IAC + DONT + opt)
    except EOFError: # raised by self.rawq_getchar()
        self.iacseq = '' # Reset on EOF
        self.sb = 0
        pass
    self.cookedq = self.cookedq + buf[0]
    self.sbdataq = self.sbdataq + buf[1]
telnetlib.Telnet.process_rawq = _process_rawq

then override the Telnet class' method:

telnetlib.Telnet.process_rawq = _process_rawq

This solved the problem for me.

Fredrik Johansson · Answer 2 · 2015-09-15T18:58:56.483

This code (http://www.opensource.apple.com/source/python/python-3/python/Lib/telnetlib.py) seems to just ignore null characters. Is that really correct behavior?

def process_rawq(self):
    """Transfer from raw queue to cooked queue.

    Set self.eof when connection is closed.  Don't block unless in
    the midst of an IAC sequence.

    """
    buf = ''
    try:
        while self.rawq:
            c = self.rawq_getchar()
            if c == theNULL:
                continue
:
:

process_rawq is then in turn called by e.g. read_until

def read_until(self, match, timeout=None):
    """Read until a given string is encountered or until timeout.

    When no match is found, return whatever is available instead,
    possibly the empty string.  Raise EOFError if the connection
    is closed and no cooked data is available.

    """
    n = len(match)
    self.process_rawq()
    :
    :

I also want to receive the null character. In my particular case it marks the end of a multiline message.

So the answer seems to be that this is expected behavior as the library code is written.

FWIW https://support.microsoft.com/en-us/kb/231866 states:

Communication is established using TCP/IP and is based on a Network Virtual Terminal (NVT). On the client, the Telnet program is responsible for translating incoming NVT codes to codes understood by the client's display device as well as for translating client-generated keyboard codes into outgoing NVT codes.

The NVT uses 7-bit codes for characters. The display device, referred to as a printer in the RFC, is only required to display the standard printing ASCII characters represented by 7-bit codes and to recognize and process certain control codes. The 7-bit characters are transmitted as 8-bit bytes with the most significant bit set to zero. An end-of-line is transmitted as a carriage return (CR) followed by a line feed (LF). If you want to transmit an actual carriage return, this is transmitted as a carriage return followed by a NUL (all bits zero) character.

and

Name     Code   Decimal Value   
Function NULL   NUL 0   No operation

Python reading until null character from Telnet

2 Answers2

Linked