I've written a python script using scrapy to crawl a site, and I'm trying to set up a job through jenkins to call the script nightly (this way it's very easy to see the output).
The machine I'm running jenkins on is a bitnami VM inside google compute.
I set up the command to run through the shell command in jenkins, and it's failing with the following error:
Building on master in workspace /opt/bitnami/apps/jenkins/jenkins_home/jobs/Scrape and Import myco/workspace [workspace] $ /bin/sh -xe /opt/bitnami/apache-tomcat/temp/hudson4165433582945317339.sh
+ /usr/local/myco/myscript.py -l /usr/local/myco/results/7.log -o /usr/local/myco/results/7.json -s /usr/local/myco/results/7.stats myspider
Traceback (most recent call last): File "/usr/local/myco/myscript.py", line 5, in <module>
from twisted.internet import reactor File "/usr/local/lib/python2.7/dist-packages/twisted/internet/reactor.py", line 38, in <module>
from twisted.internet import default File "/usr/local/lib/python2.7/dist-packages/twisted/internet/default.py", line 56, in <module>
install = _getInstallFunction(platform) File "/usr/local/lib/python2.7/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction
from twisted.internet.epollreactor import install File "/usr/local/lib/python2.7/dist-packages/twisted/internet/epollreactor.py", line 24, in <module>
from twisted.internet import posixbase File "/usr/local/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 23, in <module>
from twisted.internet import error, udp, tcp File "/usr/local/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 29, in <module>
from twisted.internet._newtls import ( File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 21, in <module>
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 41, in <module>
from OpenSSL.SSL import Error, ZeroReturnError, WantReadError File "/usr/local/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import rand, crypto, SSL File "/usr/local/lib/python2.7/dist-packages/OpenSSL/rand.py", line 11, in <module>
from OpenSSL._util import ( File "/usr/local/lib/python2.7/dist-packages/OpenSSL/_util.py", line 7, in <module>
binding = Binding() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 114, in __init__
self._ensure_ffi_initialized() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 126, in _ensure_ffi_initialized
cls._modules, File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/utils.py", line 31, in load_library_for_binding
lib = ffi.verifier.load_library() File "/usr/local/lib/python2.7/dist-packages/cffi/verifier.py", line 97, in load_library
return self._load_library() File "/usr/local/lib/python2.7/dist-packages/cffi/verifier.py", line 207, in _load_library
return self._vengine.load_library() File "/usr/local/lib/python2.7/dist-packages/cffi/vengine_cpy.py", line 155, in load_library
raise ffiplatform.VerificationError(error) cffi.ffiplatform.VerificationError: importing '/usr/local/lib/python2.7/dist-packages/cryptography/_Cryptography_cffi_e7d09016xc302a38b.so': /usr/local/lib/python2.7/dist-packages/cryptography/_Cryptography_cffi_e7d09016xc302a38b.so: symbol EC_GFp_nistp521_method, version OPENSSL_1.0.1 not defined in file libcrypto.so.1.0.0 with link time reference Build step 'Execute shell' marked build as failure Finished: FAILURE
I'm perplexed because when I run the same command (as my user and the user jenkins is running under, tomcat) I don't get this error, the script works fine.
I suspect this may have to do with the script being executed inside apache, but I'm at my wits end and googling hasn't turned up any obvious solutions.
Any idea as to how to solve this?
