How can I display out-of-range ascii characters?

Question

I'm connected to a Linux machine using PuTTY.

On the Linux machine, I'm running a python script that takes a list of characters and prints each character, together with its index, in order.

Some of the characters in my list fall outside the range of printable ascii characters. These irregular characters are corrupting my output. Sometimes they simply don't appear, while other times they actually delete large chunks of valid text.

I thought I could correct this by turning off buffering, but the problem still occurs when I run the script using the python -u flag.

Interestingly, this problem does not occur when I pipe my input to the less reader. In less, irregular characters show up like this: <A9>, <A7>, ^V, ^@, etc. No chunks of text are missing.

I'm not sure where my problem lies. Is there a way to configure my terminal so that unpiped output will still show irregular characters?

Answer 1

You have less configured to display raw characters as ASCII character combinations representing these non-printable characters; the -r or --raw-control-chars command line switch does this:

Causes "raw" control characters to be displayed. The default is to display control characters using the caret notation; for example, a control-A (octal 001) is displayed as "^A". Warning: when the -r option is used, less cannot keep track of the actual appearance of the screen (since this depends on how the screen responds to each type of control character). Thus, various display problems may result, such as long lines being split in the wrong place.

This is a special less feature.

If you want to do the same with your Python program, you'll need to make the same translation yourself. Create a mapping turning 'special' characters into escape codes:

nonprintable = {
    '\x00': '^@',
    '\x01': '^A',
    '\x02': '^B',
    '\x03': '^C',
    '\x04': '^D',
    # etc.
}

for i in range(128):
    character = chr(i)
    print i, nonprintable.get(character, character)

Answer 2

is the remote machine maybe set to unicode (modern linux distros are), then make sure you are running putty with the unicode setting too.