2

I currently have a couple sites setup on a server, and they all use beanstalkd for their queues. Some of these sites are staging sites. While I know it would be ideal to have the staging site on another server, it doesn't make financial sense to spin up another server for it in this situation.

I recently ran into a very confusing issue where I was deploying a staging site, which included a reseeding of the database. I have an observer setup on some model saves that will trigger a job to be queued, which usually ends up sending out an email. The staging site does not actually have any queue workers setup to run them, but the production site (on the same server) does have the queue workers running.

What appeared to be happening is the staging site was generating the queue jobs, and the production site was running those queue jobs! This causes random users of mine to be spammed with email, since it would serialize the model from staging, and when it unserialized it running the job, it actually matched up with an actual production user.

It seems like it would be very common to have multiple sites on a server running queues, so I'm curious if there is a way to avoid this issue. Elasticsearch has the concept of a 'cluster', so you can run multiple search 'clusters' on one server. I'm curious if beanstalkd or redis or any other queue provider have this ability, so we don't have crosstalk between completely separate websites.

Thanks!

Andrew Brown
  • 5,330
  • 3
  • 22
  • 39

2 Answers2

3

Beanstalkd has the concept of tubes:

Tubes are job queues.

A common use case of tubes would be to have completely different sets of producers and consumers running through a single beanstalk instance such that a given consumer will not know what to do with jobs produced by some of the producers. Producer1 can enqueue jobs into Tube1 and Consumer1 can pick up the jobs from there completely independently of what Producer2 and Consumer2 are doing with Tube2, for example.

For example, if you're using pheanstalk, the producer will call useTube():

$pheanstalk = new Pheanstalk();
$pheanstalk->useTube('foo')->put(...);

And the worker will call watch():

$pheanstalk = new Pheanstalk();
$pheanstalk->watch('foo')->ignore('default')->reserve();
Alex Howansky
  • 50,515
  • 8
  • 78
  • 98
  • as I understand it, tubes are more of a way to separate responsibilities **within** a project. for example, you could have an email tube, SMS tube, database notification tube, etc. what I need is separation between projects, and really the projects shouldn't know anything about each other, so they should both be okay having an `email` queue. – Andrew Brown Apr 17 '17 at 20:10
  • Beanstalk doesn't have any clue what a "project" is. You can use them however you like. – Alex Howansky Apr 17 '17 at 20:12
  • _"so they should both be okay having an email queue"_ Then just use the project name (or server name) as part of the tube. E.g., `foo-email` and `bar-email`... – Alex Howansky Apr 17 '17 at 20:13
  • yah, that was going to be my hacky way of handling this, cause now I'll need `foo-production-email` and `foo-staging-email`. I was hoping they had a concept similar to `clusters`, or essentially like a namespace for the project. – Andrew Brown Apr 17 '17 at 20:17
  • It's not hacky, that's how beanstalk works. As long as you don't push jobs into the default queue, every project will be isolated. Depending on your system design, you may decide you want to run separate beanstalk instances for dev/staging/production, but running different instances for separate projects in those environments is just overkill. – Alex Howansky Apr 17 '17 at 20:21
0

This is an old question but have you tried running multiple beanstalkd daemons? Simply bind to another port.

Example: beanstalkd -p 11301 &

Use & to fork into background.

Andi Li
  • 1