Removing all non-printable ASCII UTF-8 characters in a text file

Question

I have a text file that I'm trying to view in notepad++. The file contains a list of reptile subspecies names in one column, and then I have DNA sequence IDs that are separated by what I thought was just white space. But when I open the text file, it appears that the space is occupied with unprintable characters such as GS and VT.

For example:

Subspecies name, unprintable characters, sequence ID, unprintable char, sequence ID... and so forth until the next line:

Ablepharus bivittatus GSGSGSGSGS 36660VT4560VT23400

Is there a way I can remove all of these unprintable GS and VT characters from my text file? When I try to print every line in the file, I keep getting weird spacing due to these control characters that I see in notepad++. Any way I can make it just print normally without all of the spacing disruptions from the unprintable characters?

Updates:

I used user312016's advice and installed chardet for Python. I found out the file is encoded in UTF-16LE.

I got the file from a professor off of dropbox. The file was zipped, and all I did was unzip the file. It was a .txt file and I'm sure he didn't mention using another script to parse the data.

When I click on the unzipped .txt file to open it in regular notepad, it displays weird symbols that I assume are the GSs and VTs that I see when I open the file in notepad++.

Answer 1

When encountering strange characters in a "text" file, the right thing to do is to contact whoever created the file (possibly just by reading elsewhere on their Web site) to find out what they were trying to send you. Meta-information like character encoding, let alone more complex ideas like file and record format, are mostly transmitted out-of-band, meaning at best you will find only hints of them in the file itself.

In this case, however, you might have a "plain text" file that uses some of the more obscure ASCII control codes to separate records and fields in a table.

The Group Separator you've encountered, along with its siblings, were intended to separate fields and rows (and weirder subdivisions) of ASCII text data like what you have. Here's the relevant rows from the Wikipedia chart I linked above, stripped down some:

       Python
Dec    String    Abbr  Keyboard  Name
--------------------------------------
11     '\x0b'    VT    Ctrl-K    Vertical Tab

28     '\x1c'    FS    Ctrl-\    File Separator
29     '\x1d'    GS    Ctrl-]    Group Separator
30     '\x1e'    RS    Ctrl-^    Record Separator
31     '\x1f'    US    Ctrl-_    Unit Separator

That string of Group Separators you encountered could indicate a bunch of empty groups next to each other, in the same way that a bunch of commas next to each other ('Obama,Barack,,,,44') indicate empty cells in the CSV representation of a spreadsheet. The Vertical Tabs might separate "rows" (instead of, or in addition to, one of the separators above).

But this is all just guesswork. It's just as likely this file is not "plain text" at all, but the export format of some database or spreadsheet program. Again, whoever published the data ought to have also explained the file format somewhere... If not, and if you can't contact them, then educated guesswork is all you've got.

Answer 2

You have to know in which encoding your file was encoded. Your issue comes from the fact you are decoding your file in a different and incompatible encoding as it was written on the storage device.

Then, you will just have to do something like this:

with open('file.txt', rb) as f:
    file_decoded = f.read().decode('the_encoding_of_the_file')

If you don't know the encoding, there is no way to do this reliabely. But you can still use a library that is going to try to determine the encoding like chardet.