How to resolve UnicodeEncodeError storing data in json format

Question

I have scraped data from a website but for some items it shows me below error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\2019' in position 4: ordinal not in range(128)

I have even put "# -- coding: utf-8 --" at the top of the document but it is not worked. Please help.

Possible duplicate of [UnicodeEncodeError on API-call (json)](https://stackoverflow.com/questions/45154063/unicodeencodeerror-on-api-call-json) — bgse, Nov 09 '17 at 12:02
That's not what `# -- coding: utf-8 --` is for. Show your code. — Serge Ballesta, Nov 09 '17 at 12:14
@michael . you need to encode the data while printing it or writing it to file , what you did is a way specifying the encoding of a Python file — Nishant Singh, Nov 09 '17 at 12:18
Are we supposed to guess what your code was doing at this point and what your data looks like ? — bruno desthuilliers, Nov 09 '17 at 12:27

hspandher · Answer 1 · 2017-11-09T12:25:36.967

Either always consider the unicode content or remove the unicode content entirely. The error is occurring because you (or some library methods you're using) are trying to convert utf-8 content into ascii without ignoring the errors.

# Ignore unicode content
content_string = content_string.encode('ascii', 'ignore')

# Or make sure you handle unicode content as such. It would have been
# easier if you're using Python3x.

The purpose of # -- coding: utf-8 -- is to allow explicitly adding Unicode content into a python code file, and not to set the default encoding.

# -- coding: utf-8 --
book_name = 'Les Misérables'

How to resolve UnicodeEncodeError storing data in json format

1 Answers1