Error: too many emails per second on Heroku

Question

I have rails 4 app on Heroku sending emails via deliver_later. Sidekiq is running and working and I have config.active_job.queue_adapter = :sidekiq in my config.

I also see this in the logs [ActiveJob] [ActionMailer::DeliveryJob] which makes me believe emails are being sent as background jobs.

So why do I still get errors about too many emails per second?

Net::SMTPUnknownError: could not get 3xx (550: 550 5.7.0 Requested action not taken: too many emails per second

I just noticed I have sidekiq concurrency set to 3. Maybe that's the problem?

niborg · Answer 1 · 2017-02-02T02:13:34.147

2

This is not going to be related to Sidekiq (well, not directly, anyway). This relates to whatever SMTP server you are using and the server's rate limiting. If you have multiple instances of Sidekiq firing off emails near simultaneously, this error might get thrown if you have a relatively low rate limit.

For instance, if you are using Mailtrap as a SMTP server in a development or staging environment, a free account is limited to 2 emails per second. Pretty easy to run into this error.

If you can't avoid running into this issue for your purposes, you can try a few different strategies by using the keyword argument 'wait' for ActionMailer's deliver_later method, e.g.:

users_i_need_to_email.each_with_index do |user, index|
  t = 5 * index
  UserMailer.send_important_stuff(user)
            .deliver_later(wait: t.seconds)
end

In the event your :deliver_later calls are operating in different processes or flows, you can get away with randomizing the delivery time (say, some random time up to 5 minutes).

Obviously, neither solution above is full proof, so you are best off using a SMTP server that is adequate for your purposes.

edited Feb 02 '17 at 02:13

answered Feb 02 '17 at 01:45

niborg

390
4
16

Yep, everything you say is true. My first solution was to use a random delay but it felt like a hack and I didn't like the delay. So now my solutions is to put my mailer queue into it's own dyno and set concurrency to 1. Still a hack...but feels less hacky...i guess. heh – Dty Feb 02 '17 at 19:27
Would be nice to catch errors in ActionMailer and retry -- but my quick look into the subject it seems that is not available yet. This might work though: use a Sidekiq job to process your mailer, have the job call your mailer with 'deliver_now' instead of 'deliver_later', and have Sidekiq catch these errors and retry. (Also seems unnatural :/) – niborg Feb 02 '17 at 22:29
We have something similar happening in another project. Sidekiq has been retrying the errors for over a year now without issues so we never bother to fix it -_- – Dty Feb 03 '17 at 00:10

score 1 · Accepted Answer · edited May 23 '17 at 11:45

Definitely seems to be an issue with sidekiq concurrency set to 3.

Additional info: Setting concurrency to 1 fixes the issue but I don't want all jobs to be processed by a single worker. It would be great if Sidekiq would let use set concurrency per queue so emails could use a single worker while other job could use multiple workers. However, sidekiq author said it's not possible.

I found a solution to my particular problem here using heroku which involves putting each queue in it's own heroku dyno which allows you to set concurrency in each dyno.

Error: too many emails per second on Heroku

2 Answers2