1

I'm fairly new to Python, so I'm hoping this is something simple that I'm just missing.

I'm running Python 2.7 on Windows 7

I'm trying to run a basic twitter scraping program through the command line. However I keep getting the following error:

File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u2019' in position 79: character maps to (undefined)

I understand basically what's happening here, that it's trying to print to the console in cp437 and it's getting confused by the unicode characters in the tweets that it's grabbing.

All I'm trying to do is either get it to replace those characters with "?" or just get it to drop those characters altogether. I have read a bunch of posts about this and I can't figure out how to do it.

I opened the cp437.py file that's referenced in the error and I changed all the errors='strict' to errors='ignore' but that didn't solve the problem.

I then tried to go into the C:\Python27\Lib\codecs.py file and change all the errors='strict' to errors='ignore' but that didn't solve the problem either.

Any ideas? Like I said, hopefully I'm just missing something basic but I've read a bunch of posts on this and I can't seem to puzzle it out.

Thanks a lot. Seth

seth127
  • 2,594
  • 5
  • 30
  • 43
  • 1
    You can't just "ignore" errors to make them go away. Changing built-in libraries is probably not a good idea. – KSFT Apr 19 '15 at 15:00
  • 1
    You should fix your code. Without more information, I can't be more specific. – KSFT Apr 19 '15 at 15:01
  • http://stackoverflow.com/questions/3224268/python-unicode-encode-error possible duplicate – pnv Apr 19 '15 at 15:16

1 Answers1

3

I would not suggest changing the built in libraries - they are designed to allow handling encoding errors without needing to be fiddled with (and if you have change, not longer clear that any solution that would work for everyone else, would work for you).

You may just want to be passing errors='ignore' into whatever encoding function you are using to just skip the error character, or errors='replace' to replace that character with the character \ufff to signify there was a problem. [ error='strict' is the default if you don't pass any value. ]

However, if you are printing to the command line, you probably don't want to be encoding as unicode anyway, but ASCII instead - since unicode includes characters that the command line can't print. (and i suspect that that which is causing errors to be thrown up, rather than there being non-standard unicode characters in the response you are getting from Twitter).

Try e.g.

 print original_data.encode('ascii', 'ignore')
kyrenia
  • 5,431
  • 9
  • 63
  • 93
  • Thanks. That was a much easier solution. I was just down a rabbit hole that I needed to stay out of. – seth127 Apr 19 '15 at 16:38