Celery + SQS - pycurl error

Question

Today I have been trying to setup Celery using AWS SQS as the broker, however upon excututing the following:

test.py

from celery import Celery

access_key_id = '********************'
secret_access_key = '****************************************'

broker_url = 'sqs://%s:%s@' %(access_key_id, secret_access_key)

app = Celery('test', backend=None, broker=broker_url)

@app.task
def add(x, y):
    return x + y

Attempting to run the Celery worker using the command

celery -A test worker --loglevel=info

With this I get the following error:

Unrecoverable error: ImportError('The curl client requires the pycurl library.',)

I have the following packages installed:

amqp (2.2.2)
billiard (3.5.0.3)
boto (2.48.0)
boto3 (1.4.7)
botocore (1.7.48)
celery (4.1.0)
docutils (0.14)
jmespath (0.9.3)
kombu (4.1.0)
pip (9.0.1)
pycurl (7.43.0)
python-dateutil (2.6.1)
pytz (2017.3)
s3transfer (0.1.11)
schedule (0.5.0)
setuptools (37.0.0)
six (1.11.0)
vine (1.1.4)
wheel (0.30.0)

Can anyone see what I am doing wrong? Thanks in advance!

score 5 · Accepted Answer · answered Nov 22 '17 at 01:29

5

Looks like it could be an issue with the version of Celery (4.1.0) that you have. If I execute the same code after downgrading to version 3.1.25 it works fine.

answered Nov 22 '17 at 01:29

Sudhi Pulla

574
10
19

1

I tried this and it does fix this particular issue, but it leads to other compatibility issues (in my case with the version of Django), so not sure if this is the ideal solution. – kashgo Dec 09 '20 at 18:34

score 4 · Answer 2 · edited May 17 '22 at 17:56

4

I had the same error and these steps worked for me:

mac:

brew upgrade openssl

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"   

export PYCURL_SSL_LIBRARY=openssl

python -m pip install celery[sqs]

centos:

pip uninstall pycurl
export PYCURL_SSL_LIBRARY=nss
pip install --compile pycurl

edited May 17 '22 at 17:56

UrbanConor

162
1
5
16

answered Apr 27 '20 at 07:52

BlaShadow

11,075
7
39
60

Worked for me on Mac! – kashgo Dec 09 '20 at 18:46
This almost worked, but I would (explicitly) say that you need to uninstall pycurl and then reinstall after following those steps. Thank you! – Jordan The Genius Apr 26 '22 at 22:16
Note that the LDFLAGS and CPPFLAGS paths depend on where your openssl is installed. Run `brew info openssl` to find the paths. – Petr Gazarov May 26 '23 at 07:03

score 3 · Answer 3 · edited May 12 '22 at 14:10

If you are on Mac, you need to compile and install PycURL. You may also need to install openssl if not already installed.

brew install openssl
    
export PYCURL_SSL_LIBRARY=openssl 
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include" 
pip install --no-cache-dir pycurl

Similarly for linux to install openssl:

apt-get update -y && apt-get install -y libcurl4-openssl-dev libssl-dev gcc

Check the PycURL SSL Docs for further installation info.

Sascha Rau · Answer 4 · 2021-02-24T11:15:37.760

0

This works on Debian

sudo apt install libcurl4-openssl-dev libssl-dev
pip uninstall pycurl
pip install pycurl --compile --global-option="--with-openssl" --no-cache-dir

edited Feb 24 '21 at 11:15

answered Feb 23 '21 at 15:04

Sascha Rau

357
3
12

Celery + SQS - pycurl error

4 Answers4