We have a php app using laravel on elastic beanstalk that's been running for almost 2 years. Since the last 2 weeks(almost 3) we've been having random really high response time which cause the Database connections to become stuffed which then cause the app to crash due to too many connections. The only changes to the code prior to the first time it happened were a month before and were only minor changes.
When this happens everything in the monitoring section of beanstalk is staying stable except for the target response time which can go up to 40 seconds. Since nothing can be done with the server when it goes to this point (everything just times out) I have no choice but to redeploy the same version of the app to reset the server, after that we're good to go for a day or 2 before it happens again.
I've implemented Laravel telescope to see if I could find out anything about the requests coming in when it's happening but all I can see is that some pretty simple requests starts to take from 4 to 150 seconds for no apparent reason. Also when the app is stable I've noticed that telescope is logging some slow database query that takes up to 200ms, when I tried these query directly in mySQL workbench it took around 31ms.
Telescope is only logging request that takes more than 3 seconds, maybe I should change that to log everything? But we have a pretty big traffic so I feel like it's gonna be hard to find anything in that. I'm not sure what to do at this point. What would you guys do to debug this?
