Ok, so here is our setup:
We have 3 web-dynos running on Heroku with our rails application (rails with puma - we also tested unicorn). This app is connected to a postgresql DB which is quite strong (max pool_size of 120). We also tested 2 or 4 dynos
We have about 800 requests per minute hitting this API server.
Sometimes, we are getting quite large response times. The strange thing is, that it only happens on 1 web-dyno. At least this is what we can see in our logs.
The other 2 dynos are just about perfect in response time for the same route (during the time the other dyno is having huge response problems). This goes on for a couple of minutes until its back to "normal". Until its showing the same behaviour again a couple of hours later.
We can't find any pattern that would help us find out what to do about it.
Does anyone have an idea of how we can attack that problem?
Thanks a lot in advance.