How to set up celery and django with Amazon SQS

Question

I'm trying to convert videos uploaded by users in a django app. Problem is, it's eating up resources and the site becomes unavailable during this time and a new instance has to be sprung up to scale it(I'm using Elastic Beanstalk). I did some research and decided to use SQS with a worker enviroment

I set up celery and added necessary configs to the settings.py file

BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
    'region': 'us-east-1',
    'polling_interval': 3,
    'visibility_timeout': 3600,
}
BROKER_USER = AWS_ACCESS_KEY_ID
BROKER_PASSWORD = AWS_SECRET_ACCESS_KEY

CELERY_DEFAULT_QUEUE = 'celery-convert-video'
CELERY_QUEUES = {
    CELERY_DEFAULT_QUEUE: {
        'exchange': CELERY_DEFAULT_QUEUE,
        'binding_key': CELERY_DEFAULT_QUEUE,
    }
}

I set up the POST url to /celery-convert-video/ I also do:

video = TestVideo.objects.create(uploaded_video=videoFile)

then

ConvertVideo.delay(video_id=video.id)

to send the task to SQS. It grabs he uploaded file using a url and converts it. Locally it works but the problem comes in the cloud.

I seem to be having problems with setting up the Worker Environment because it's health always ends up becoming "Severe" and everytime I check the cause It says 100% of requests return error 4xx . Logs say it's a 403.

The tasks show up fine in SQS (but are indecipherable, just random letters, I'm guessing it's encoded?) and get "in flight" so the problem I'm assuming is the worker. I have no idea how to set it up properly.

So a few questions:

Do I have to use the same deployment files for the worker and the main enviroment?
Do I have to edit the deployment files and add a /celery-convert-video/ view in my worker environment and then send it to the equivalent ConvertVideo function in the worker?
How do I connect the worker to the RDS database used by the main enviroment?
How do I get rid of the 403 errors and get the worker health back to green?

If anyone has a step by step tutorial or something that they can point me to that would be a big help!

P.S. I'm not an expert in cloud computing, It's actually my first stab at it so forgive my ignorance..

How to set up celery and django with Amazon SQS

0 Answers0