I am on python 2.6
HTTPD 2.2.15
CentOS 6
I am writing a python script that involves making a lot of REST API calls. I decided to try increase the speed of the program by multi-threading the URL requests. I used the requests module and everything worked fine. I had to switch to pycurl so I copied pycurl code I had that works.
Now when I run the code I get errors in some threads sometimes. Not the same threads everytime. By commenting out all the threads except 1 I cannot reproduce the error, but all it takes is 2 or more threads for the errors to potentially show up.
I can't share the exact code, but here is an approximation:
import os
import datetime
import time
import pycurl
import threading
from StringIO import StringIO
from pprint import pprint as pprint
import json
variable1 = {}
variable2 = {"x": {}, "y": {}}
def pycurl_method(url, creds):
buffer = StringIO()
c = pycurl.Curl()
c.setopt(c.URL, url)
c.setopt(c.USERPWD, creds[0] + ':' + creds[1])
c.setopt(c.WRITEFUNCTION, buffer.write)
c.perform()
c.close()
response = buffer.getvalue()
return json.loads(response)
def url_method(url, creds):
env = threading.current_thread().getName()
response = pycurl_method(url, creds)
...Do stuff...
def start_threads(threads):
for thread in threads:
thread.start()
def join_threads(threads):
for thread in threads:
thread.join()
def main():
apiUrl0 = "some URL"
apiUrl1 = "some different URL"
apiUrl2 = "etc"
...
creds = #returns credentials from a file stored on the server as [username,password]
#Multithread API calls
t0 = threading.Thread(target=url_method, name='nameOfThread0', args=(apiUrl0, creds))
t1 = threading.Thread(target=url_method, name='nameOfThread1', args=(apiUrl1, creds))
t2 = threading.Thread(target=url_method, name='nameOfThread2', args=(apiUrl2, creds))
...
start_threads([t0, t1, t2, ...etc...])
join_threads([t0, t1, t2, ...etc...])
Errors are like this:
File "this_script.py", line xyz, in url_method
c.perform()
error: (77, 'Problem with the SSL CA cert (path? access rights?)')
If I remove the threads and execute sequentially, I get no errors.
I think It might be related to pycurl (since requests was fine).
Specifically, pycurl.Curl().
I wonder if each thread is somehow using the same pycurl.Curl() and stepping on the toes of other threads.
Why do these errors show up?
How can I avoid them?
Do the threading and pycurl modules not play nicely together?
