4

Sidekiq documentation says:

Don't set the concurrency higher than 50. I've seen stability issues with concurrency of 100, for example

Well, my low memory consumption enables me to use concurrency of 350 threads on a single 512MB X1 heroku dyno. And I would like to use ~300 because all jobs are IO intensive (http requests).

I wonder what issues can I encounter in?

I tried to monitor the logs at overload with 80 and seen no issues.

What issues should I expect when setting up concurrency of 300 threads? Will I risk jobs getting terminated without being moved to the "dead" queue? OR just a termination of workers that I will be able to watch. Is it safe to set a concurrency of 300 or 100?

The owner of sidekiq doesn't know the answer and here is the issue I opened.

UPDATE: In high load, when I increased from 80 to 100 I started getting 'can't create Thread: Resource temporarily unavailable' errors here and there, in extreme cases of 180 threads it will sometime terminate the entire sidekiq process.

The memory consumption was always between 140MB to 240MB according to Heroku metrics.

I used TTIN signal as describe here

And found that most threads are waiting on those lines of code:

app[worker.1]: 3 TID-ow5z46exw WARN: /app/vendor/ruby-2.3.0/lib/ruby/2.3.0/monitor.rb:187:in `lock'

app[worker.1]: 3 TID-os9ulw8ps WARN: /app/vendor/ruby-2.3.0/lib/ruby/2.3.0/net/http.rb:880:in `initialize'

app[worker.1]: 3 TID-os9ulw8ps WARN: /app/vendor/ruby-2.3.0/lib/ruby/2.3.0/timeout.rb:95:in `join'

app[worker.1]: 3 TID-osjnd6zac WARN: /app/vendor/ruby-2.3.0/lib/ruby/2.3.0/net/protocol.rb:158:in `wait_readable'

Everything is documented in the github issue

The owner of sidekiq says that the traces looks fine, so no luck spotting the root cause for the stablity issue, but there is input in how many threads causes it and what is the symptom.

naviram
  • 1,445
  • 1
  • 15
  • 26

2 Answers2

2

Based on the number of concurrency the DB pool size has to be increased. concurrency (thread) + 2 = DB connection pool size (300+2) = 302 DB connection

The real concurrency for a single sidekiq process depends on the number of processor core and other parameter. So using more currency will take most of the time in thread context switching instead of doing real IO/computation.

512MB X1 heroku dyno

Normal rails app will need atleast 200 MB of memory at startup and if each thread takes (approx) 1 MB of memory then the total consumption of memory will be

200 + (300 *1) = 500 MB

If some thread need more memory during computation (i.e, fetching more ActiveRecords, reading a large file, ...), then the whole sidekiq will crash.

When I ran sidekiq with full machine potential, garbage collection is doesn't happening immediately, which caused the memory to increase and crashed the sidekiq freqently.

Also with more threads, the completion of each job takes more time than the usual time. Analyze this case in your environment.

Sivalingam
  • 841
  • 11
  • 16
  • Sivalingam - everything is true and solid, btw I am avoiding using rails when possible and use plain ruby to reduce the memory consumption, it enabled me to use 2X34 puma threads. And much more than 300 in sidekiq. But everything that you said here is resource limitation and performance issues which are easy to monitor and tune. The owner of sidekiq talks about stability issues, I am looking for those stability issues. Running out of memory and connection, sends you a clear message. – naviram Oct 29 '16 at 21:12
  • OK, after increasing the threads further to 160, the maximum memory is only 150MB in the 24h period. But I did get this error 'can't create Thread: Resource temporarily unavailable' that claimed here (https://github.com/mperham/sidekiq/issues/286) to be caused due a thread leak that should be monitored with this https://github.com/mperham/sidekiq/wiki/Signals#ttin. I will monitor it and update the post. – naviram Oct 29 '16 at 21:14
2

Well, sidekiq stability issues in high concurrency are as follows.

When you are setting a concurrency that is higher than 80 (or 50) you may encounter in this error "can't create Thread: Resource temporarily unavailable:"

Some jobs will return back to queue, sometimes the entire process will be terminated and jobs will be lost, unless you use sidekiq pro reliability feature

It seems that we are hitting heroku's maximum 256 threads limitation although sidekiq is configured to use 80 threads. It doesn't help if I use multiple sidekiq processes inside single heroku dyno when I did it, I still ran into this limit.

It seems like a thread leak, and this is the next thing to investigate.

The above will happen also when the memory consumption will stay low (< 240MB in my example)

Everything is updated in the github issue

Community
  • 1
  • 1
naviram
  • 1,445
  • 1
  • 15
  • 26