0

I am having problem on running my crawler

UnicodeEncodeError: 'ascii' codec can't encode characters in position

I am using this code 

author = str(info.css(".author::text").extract_first())

but still I am having that error any idea how can solve it? Thank you!

Here's the error

Traceback (most recent call last):
 File "/usr/local/lib/python2.7/site-packages/scrapy/utils/defer.py", line 
 102, in iter_errback
yield next(it)
  File "/usr/local/lib/python2.7/site-packages/sh_scrapy/middlewares.py", line 30, in process_spider_output
for x in result:
 File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
 File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
 File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
 File "/usr/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
  return (r for r in result or () if _filter(r))
 File "/app/__main__.egg/teslamotorsclub_spider/spiders/teslamotorsclub.py", line 40, in parse
author = str(info.css(".author::text").extract_first())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)
Christian Read
  • 135
  • 11

1 Answers1

1

Try:

author = info.css(".author::text").extract_first().decode('utf-8')

The reason for this is extract_first returns a raw bytes object. To convert this to a string, python makes no guesses as to how it's encoded, therefore, you need to make that explicit. Utf-8 will handle just about anything you throw at it.

hd1
  • 33,938
  • 5
  • 80
  • 91
  • Thanks for helping me but still I got the error return codecs.utf_8_decode(input, errors, True) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128) – Christian Read Apr 25 '19 at 15:24
  • Post your entire script and I'll be able to help further – hd1 Apr 25 '19 at 17:12