0

I'm running a celery + rabbitmq app. I start up a bunch of ec2 machines, but I find that my celery worker machines only use about 15% cpu (peak of 30%). I've configured 6 celery workers per machine ( I just set number of workers to a big higher than the virtual cpu #).

Shouldn't celery workers be close to using 100% cpu utilization? I feel as if my workers are "slacking off" and was wondering if this is a common occurrence. Should I just leave the default 1 celery worker per core?

When looking at my celery workers logs, they all are doing work and connecting to my single broker successfully. I have turned off options such as celery heartbeats because they were giving me syncing/delay problems.

MORE INFO: I am not using the celery --concurrency option or eventlet even though I am using multiple workers.

jeffrey
  • 3,196
  • 7
  • 26
  • 44
  • 2
    Do you have a lot of I/O? Because in most cases my processes don't use 100% of the CPU because they are waiting for something. Maybe you could check the server load instead of CPU. If there is one CPU, a load of 1 or more means you are using the max. of that machine. – Alexis Wilke Oct 19 '14 at 06:56
  • Ah, well my task performs a regex scan on a file from S3, so it has to pull the file from S3. It also has to open a file very task. That would be the I/O you're talking about, correct? Is there any way I can speed up the process then? Each of my tasks takes about .5-.8s – jeffrey Oct 19 '14 at 22:56
  • I'm not familiar enough with the Amazon network to tell you how you could (if possible) optimize access to an S3 storage. But it is not unlikely that by loading a file through the network is something beyond your ability to make it go faster. Depending on the amount of network I/O used, you could duplicate your processes on a single node to spend less on "hardware". But that will not resolve your current speed issue... – Alexis Wilke Oct 19 '14 at 23:50

0 Answers0