3

I am using beanstalkd in a Laravel project to handle jobs on a queue. Beanstalkd is running locally. What I want to do is add one or more remote servers to handle some jobs when the queue gets bigger. I know that with Laravel I can send a job to a specific remote connection but in this way I don't know the load in each server prior to sending the job.

I was wondering if beanstalkd supports load balancing between servers and error handling when a remote job fails for example.

Thank you

nteath
  • 250
  • 1
  • 13

1 Answers1

4

Beanstalkd does't have features for load balancing.

You could setup a HAProxy on your balancer and signup multiple servers with beanstalkd installed. Then when you send jobs from Laravel code you send to the HAProxy, and HAProxy decides on which sub-server puts the job, as it knows the loading and if there is an incident with a sub system.

In the code you just need to change the IP. In your infrastructure you need to have balancer (HAProxy) that is setup with a pool of Beanstalkd servers.

We usually have 2 machines, and they are configured like this:

- Machine 1: HAProxy, Apache, MySQL, Laravel, Beanstalkd
- Machine 2: MySQL, Laravel, Beanstalkd
Pentium10
  • 204,586
  • 122
  • 423
  • 502
  • Thank you for your answer. I managed to setup an infrastructure and the load is being balanced through the HAProxy. I was wondering if there is any way to get feedback from the remote servers on the job status. If it completed successfully or an error occurred. When I had beanstalkd only locally, part of the job was to write the result to the database. I want to avoid this now and the remote servers will not have access to database. – nteath Sep 01 '16 at 14:36
  • You should post as a new question, but since it's async, you cannot be connected and wait until it's processed. What you can do is that in the worker who handles the job add code that writes somewhere the status of the job. It could be Memcache, Redis, or another tube, a response tube, that was given in the job message. Use case: on a UI user wants to preview an pdf layout, the UI generates a tube name+server, and starts to watch it, meanwhile puts the job to beanstalkd(via haproxy), one of the workers picks up the job, resolves, and writes the answer to the server/tube given as input param – Pentium10 Sep 01 '16 at 17:55
  • so now the answer is on the tube that was given by the param, and we already know that there is a watcher the UI on it, so the UI will pick up the answer and deal with it. This way you keep the UI busy and connected, you bypass the async way, but you have a clear monitoring. – Pentium10 Sep 01 '16 at 17:56
  • Easier would be if you keep a flag somewhere in a system and read it later from there. – Pentium10 Sep 01 '16 at 17:57
  • 1
    But how do you consume jobs? Do you connect trough hapxoy or do you connect to each beanstalkd server and fetch jobs? – user237329 May 10 '17 at 08:46