2

I have a celery task like this:

@celery.task
def file_transfer(password, source12, destination):
    result = subprocess.Popen(['sshpass', '-p', password, 'rsync', '-avz', source12, destination], 
                                    stderr=subprocess.PIPE, stdout=subprocess.PIPE).communicate()[0]                             
    return result

I have called in a Djagno view.

User can select more than one file to copy to the destination. For example if the user selects, 4 files at once, celery accept only 2 tasks. What's wrong?

pynovice
  • 7,424
  • 25
  • 69
  • 109

1 Answers1

1

Have you checked the concurrency setting of your worker ?

If for example you have only one worker running on a two-core machine, concurrency by default will be 2. Which means only two tasks can be executed at once.

You can change this setting from the worker command-line with the switch:

 -c N

where N is number of parallel tasks

mpaf
  • 6,597
  • 6
  • 38
  • 42
  • Where's the concurrency setting and can you please give me the full command to change the setting. – pynovice Mar 11 '13 at 09:03
  • I did that but I am getting this error: consumer: Cannot connect to amqp://guest@127.0.0.1:5672//: [Errno 111] Connection refused. Trying again in 6.00 seconds... I am using database backend. – pynovice Mar 11 '13 at 09:24
  • the amqp is still needed for processing the messages. The database backend you are setting is probably for the results. For the queueing system you should use the BROKER_URL="django://" setting. See here: http://docs.celeryproject.org/en/latest/getting-started/brokers/django.html#broker-django – mpaf Mar 11 '13 at 10:22
  • You get that error because your BROKER is still set to amqp:// and you probably have no instance of AMQP running in localhost:5672 – mpaf Mar 11 '13 at 10:23
  • I have this: BROKER_URL = "sqla+mysql://root:password@localhost/galaxy" – pynovice Mar 11 '13 at 10:43
  • and what do you have for CELERY_RESULT_BACKEND? If it is anything like "amqp://" then you still need a running amqp server. The default is 'amqp' by the way so you should change it to the same as BROKER_URL if you want to use the mysql database for result storing. – mpaf Mar 11 '13 at 12:34