2

The actual problem

I am using Rails 3.0.4 and MongoDB through mongoid.

I am using @mailgun to deliver my emails. I send both bulk(like news-letter) and transactional(like activate account, forgot password) emails. Right now, I am using a single domain(translates to single queue on mailgun's end) to deliver those mails. Problem arises when I have lot of bulk emails already queued up and somebody registers or requests for a new password. I want the transactional emails to be delivered before the bulk mails, but mailgun queue works on a FIFO basis.

I figured a way to mitigate this can be to use different domains(hence different queues which can be processed simultaneously) for bulk and transactional mails. In rails smtp settings are application level settings rather than request level settings. So, I figured, I will use different environments for different smtp setting.

I also have a queue system for mails and am using delayed_job to handle that. I can't figure out a way to differentiate between bulk and transactional mails in delayed_job. So, I decided to move my queue system to either rescue+redis or beanstalked+stalker, where I can tag queues and can ask a worker to only process a particular queue.

The question

I want something which is easier to maintain, least resource hungry and can scale well.

  • With delayed_job I needn't run any other server and monitor it.
  • For delayed_job I was using 256MB slice on rackspace, but redis and stalker would require another server, either resque or beanstalkd.
  • I have no idea about scaling, but its 2nd month since my app launched and I have already sent 30k+ emails.

If there any alternatives to porting from delayed_job to redis or stalker, please let me know.

Update:

Seems like delayed_job also have support for named queues now, but it is not documented yet. Opening a ticket to add documentation, will update with details, once I know how to use them :)

rubish
  • 10,887
  • 3
  • 43
  • 57
  • Separate @mailgun instances? One for bulk, one for transactional? – jefflunt Jul 21 '11 at 22:28
  • Are you suggesting something or asking? If asking its like sub-users in sendgrid, which starts with a $80 plan, but on mailgun you get it with $19 plan. – rubish Jul 21 '11 at 22:32

1 Answers1

2

Delayed job accepts an optional priority parameter (it's explained down near the end of the linked page).

For example:

Delayed::Job.enqueue(MailingJob.new(params[:id]), 3)

...where 3 is the priority.

So, when en-queuing your mass-mailing, don't specify a priority, and when en-queuing your transactional emails give them something higher. This way, your transactional emails will get entered into the @mailgun queue ahead of anything that isn't already on its way out.

Unless your outgoing SMTP server has some kind of slow connection, it'll probably send several hundred emails per minute, so I wouldn't worry too much if there are, for example, 200 mass-mailings already handed off to @mailgun and you're waiting on a transactional email; it will still send shortly thereafter.

jefflunt
  • 33,527
  • 7
  • 88
  • 126
  • 1
    I know about the priority parameter, and tried to use that, but mailgun accepts emails at much higher rate than the delivery rate. So it doesn't really helped. As per homepage of delayed_job on github, "Lower numbers have higher priority." – rubish Jul 21 '11 at 22:09
  • 1
    How many emails, and what are your real-world delays, out of curiosity? Are you actually watching @mailgun's SMTP queues and making sure the delay is there (and not on the receiving end)? If so, and your queue times are still really high, I guess my answer becomes irrelevant. – jefflunt Jul 21 '11 at 22:16
  • 1
    I send weekly news-letter to all my users, right now there are nearly 3k users and my mails get delivered around 40/min and last time I ran the task I pushed all my emails to mailgun in less than 5 mins. News letters are the biggest spike in my mail traffic. There are others, but all these are of lesser magnitude than news letter. – rubish Jul 21 '11 at 22:29
  • 1
    Can you submit the mass mailings in 100-email chunks, leaving a small interval between when transactional emails will get processed, but getting your mass mailings out fast? That is, only submit the Delayed job stuff 100 elements at a time? – jefflunt Jul 22 '11 at 15:58
  • This sounds like a good idea, any pointer on how I can do this? – rubish Jul 22 '11 at 22:13
  • I'm not sure exactly. You would need some kind of counter that keeps track of the mass mailing emails, and only allow a certain number of these emails in the queue at a time, preventing more than 100 or so being submitted to mailgun at a time, without also checking, or allowing the submission of, transactional emails. – jefflunt Jul 24 '11 at 02:57
  • Thanks, for your help. Although I think eventually I would go on the route of rescue/stalker, or alter delayed_job such that it can handle multiple queues independently. – rubish Jul 24 '11 at 04:07
  • Seems like master branch of delayed_job already supports named_queues, it is just not documented yet and is targeted for 3.0 release. – rubish Jul 26 '11 at 01:23