5

I receive an error when running scrapy on Raspberry Pi 3.

I have successfully installed it, but when I try to startproject or crawl with a previously created spider, I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/usr/local/lib/python3.4/dist-packages/scrapy/cmdline.py", line 9, in <module>
    from scrapy.crawler import CrawlerProcess
  File "/usr/local/lib/python3.4/dist-packages/scrapy/crawler.py", line 7, in <module>
    from twisted.internet import reactor, defer
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/reactor.py", line 38, in <module>
    from twisted.internet import default
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 56, in <module>
    install = _getInstallFunction(platform)
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction
    from twisted.internet.epollreactor import install
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/epollreactor.py", line 24, in <module>
    from twisted.internet import posixbase
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/posixbase.py", line 18, in <module>
    from twisted.internet import error, udp, tcp
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 28, in <module>
    from twisted.internet._newtls import (
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_newtls.py", line 21, in <module>
    from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
  File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/tls.py", line 65, in <module>
    from twisted.internet._sslverify import _setAcceptableProtocols
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1865, in <module>
    "ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:"
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1845, in fromOpenSSLCipherString
    SSL.SSLv23_METHOD, SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1797, in _expandCipherString
    ctx.set_cipher_list(cipherString.encode('ascii'))
TypeError: must be str, not bytes

I have no idea why I am getting this or how to fix it, please help?

SOLUTION:

Thanks everyone for the help, I got Scrapy to work on Raspberry Pi 3 in the end by following these steps:

First install virtualenv:

sudo pip install virtualenv

Then create a virtualenv and active it for Scrapy:

virtualenv scrapyenv
source scrapyenv/bin/activate

Then I ran and updated everything in there:

apt-get update
apt-get upgrade

Install all dependencies:

apt-get install libffi-dev
apt-get install libxml2-dev
apt-get install libxslt1-dev
apt-get install libssl-dev
apt-get install python-dev

Then install Scrapy

sudo pip install scrapy

I then updated my pyOpenSSL with this:

pip -vvvv install --upgrade pyOpenSSL

This created a lot of log files and took a bit of time, after that scrapy worked fine with the normal scrapy commands and I have also run a spider - all works.

Svarto
  • 603
  • 2
  • 8
  • 20
  • 1
    Could you post your spider code? Seems like ssl related issue but the error mentions type issues: "must be str, not bytes" means twisted is expecting python2 instead python3. – Granitosaurus Feb 02 '17 at 20:48
  • ok, I wrote my code and tested it in python 3.5, however, I can't even `scrapy startproject test` without getting the same error as above. So shouldn't be my spider code that is wrong, as I get the same error with just trying to start a project. – Svarto Feb 02 '17 at 20:50
  • Are you running it from virtualenvironment? If not try setting one up, installing scrapy there and make sure you have [pyOpenSSL](https://pypi.python.org/pypi/pyOpenSSL) installed too – Granitosaurus Feb 03 '17 at 05:07
  • Worked with virtualenv and upgrading pyOpenSSL, thanks! – Svarto Feb 03 '17 at 08:42

1 Answers1

0

What user are you using? i think you need use sudo. also update your ssl. looks like the ssl connection has problems..

Have you run : pip install --verbose twisted

Also update your openssl

pip -vvvv install --upgrade pyOpenSSL, please copy the output here to check if is updated

Regards

Jhovanny Uribe
  • 218
  • 2
  • 10
  • I am running it with user pi. I ran what you suggested, unfortunately it says all requirements already satisfied.... – Svarto Feb 02 '17 at 20:51
  • have you install this : $ sudo apt-get install libffi-dev $ sudo apt-get install libssl-dev $ sudo apt-get install libxml2-dev libxslt1-dev – Jhovanny Uribe Feb 02 '17 at 21:06
  • yes, unfortunately, I also tried running those commands again and for each one it said that I have the newest version already... – Svarto Feb 02 '17 at 21:17
  • try updating your openssl, pip -vvvv install --upgrade pyOpenSSL – Jhovanny Uribe Feb 02 '17 at 21:25
  • Thanks for the help, it worked by installing a virtualenv and then running the pyOpenSSL upgrade command. This updated my pyOpenSSL and after that it worked - thanks! – Svarto Feb 03 '17 at 08:42
  • Excellent, can you write us, all the correct instructions to run Scrapy on Raspberry Pi 3?, This could help many others – Jhovanny Uribe Feb 03 '17 at 13:49