1

I've got a big project running on MRI Ruby and Rails 3.2 on Passenger with an app that was not designed with thread-safety in mind and this app handles mailings through DelayedJob and the database is paying a heavy price for this.

Some possible problems are mentioned in the sidekiq railscast http://railscasts.com/episodes/366-sidekiq including:

  1. database connection limit (if using thread-pool of 1, the db connection limit should be doubled)
  2. thread-safety (this is probably the show stopper)
  3. fiber-safety? Is this an issue with AR?

So, the question are:

  1. how viable is it to make a big project thread-safe enough for the mail generation to work in threads inside the passenger process? (mailings are complex enough to depend on AR)
  2. same question when using sidekiq, "how viable is it to make a big project thread-safe enough for the mail generation to work using sidekiq? (mailings are complex enough to depend on AR)"
  3. apart from db connection limit and thread-safety issues, is there anything else to consider or less obvious got'chas I failed to forsee?
bbozo
  • 7,075
  • 3
  • 30
  • 56
  • How many emails? Do you want to isolate the processes? – Sully Dec 15 '13 at 07:26
  • @HithamS.AlQadheeb, some web requests generate up to 3 emails, most requests that do generate mails generate just 1 mail so I'm not expecting the background threads to run for a long time after the request is delivered. Do I want to isolate the processes is a good question, it's basically behind questions 1 and 2, to go for sidekiq (separate process, decoupled from web request, but stack more infrastructure and overhead on the server), or go pure celluloid (more light-weight, but coupled with the passenger process "in sickness and in health") – bbozo Dec 15 '13 at 07:57

1 Answers1

1

I think that you are mixing up some stuff here.

I never used sidekiq, but I don't think that you have to turn on thread_safe! in your Rails application to use it.

AFAIK this is just for the request-processing to have a lock, or no lock in thread_safe! mode, so the Rails process can handle multiple requests in parallel.

When using sidekiq, that should not be one of your concerns. Just your E-Mail-Processing code needs to be thread-safe. Only the project maintainer can answer that question though.

phoet
  • 18,688
  • 4
  • 46
  • 74
  • Well.. I'm the maintainer :) thread_safe comment was just to point out that previous considerations haven't been made to make the code thread-safe (I'll edit the question to clarify this). What I'm looking to gauge is how much hurt am I in for if going for just celluloid or sidekiq to handle mailings (which use ActiveRecord to generate mails) and is it even worth it – bbozo Dec 15 '13 at 13:44
  • Nobody can answer that. It depends 100% on your application! – phoet Dec 15 '13 at 14:24
  • 1
    We were running a top 500 page in Germany without even using async e-mail sending. It's all within your setup dude. – phoet Dec 15 '13 at 14:25
  • it is kind of an answer "dont do mails asynchronously" and its certainly a viable hotfix should the DB start to break under pressure, but it's not a solution. I've got a couple of actions executing 3-4 mails to different stakeholders of the process. Tying processing of the HTTP requests (where a timely response has financial implications) to that kind of added work isn't good and will surely cause damages at some point – bbozo Dec 18 '13 at 08:56
  • i don't think that you are understanding my suggestions properly. i just want to point out that it's hard to know what is best for your application when i don't know it at all. nobody can take the work from you here, you will have to take a look on your own. – phoet Dec 18 '13 at 14:06