2

I am trying to read a PDF using Camelot library and store it to a dataframe. The resulting dataframe has garbled/bad characters in string fields.

Eg: 123Rise – Tower & Troe's Mech–

I want to remove ONLY the Garbled characters and keep everything else including symbols.

I tried regex such as these [^\w.,&,'-\s] to only keep desirable values. But I'm having to add every special character which need not be removed into this. I cannot ditch Camelot library as well.

Is there a way to solve this ??

maddy_novice
  • 49
  • 5
  • 1
    Somehow you need a list of chars what you want to replace. If you can't build it up (because there are too many or any other reason), you may want to reverse the logic and enlist the accepted chars.... – redseven Jul 07 '22 at 07:14
  • 2
    This looks like an encoding problem. Specifically, like UTF-8 text being interpreted as ISO-8859-1. Don't remove the characters, fix them – MSalters Jul 07 '22 at 07:17

4 Answers4

2

You could try to use unicodedata library to normalize the data you have, for example:

import unicodedata

def formatString(value, allow_unicode=False):
    value = str(value)
    if allow_unicode:
        value = unicodedata.normalize('NFKC', value)
    else:
        value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii')
    return(value)

print(formatString("123Rise – Tower & Troe's Mech–"))

Result:

123Rise a Tower & Troe's Mecha
Cow
  • 2,543
  • 4
  • 13
  • 25
0

One way to achieve that, is to remove non-ASCII characters.

my_text = "123Rise – Tower & Troe's Mech–"
my_text = ''.join([char if ord(char) < 128 else '' for char in my_text])
print(my_text)

Result:

123Rise  Tower & Troe's Mech

Also you can use this website as reference to normal and extended ASCII characters.

Sharmiko
  • 543
  • 4
  • 21
0

Another way I commonly use for filtering out non-ascii garbage and may be relevant (or not) is:

# Your "messy" data in question.
string = "123Rise – Tower & Troe's Mech–"

# Iterate over each character, and filter by only ord(c) < 128.
clean = "".join([c for c in string if ord(c) < 128])

What is ord? Ord (as I understand it) converts a character to its binary/ascii numeric representation. You can use this to your advantage, by filtering only numbers less than 128 (as above) which will limit your text range to basic ascii and no unicode stuff without having to work with messy encodings.

Hope that helps!

Jack Hales
  • 1,574
  • 23
  • 51
0

Removing non-ASCII characters using regex will be fast:

import re
text = "123Rise – Tower & Troe's Mech–"
re.sub(r'[^\x00-\x7F]+','', text)

The output will be:

"123Rise  Tower & Troe's Mech"
Prashant Maurya
  • 639
  • 4
  • 13