You have some performance logs, and adding more resources to the server is never a bad idea.
Things I may suggest from experience.
Check all your logs, dmesg, service logs etc, for related messages. It's quite possible in some cases for the server to have enough resources but is coming up against kernel or mysql limits... such as number of open files for example, or concurrent connections (threads) to the mysql server.
You have some monitoring running on your server, which is great, but sometimes a picture paints a thousand words, and you can identify trends more quickly by actually seeing the data in a pictorial format.
There are overwhelming amount of monitoring tools. There might even be a option in your monitoring tool output graphs of some kind.
if it transpires that mysql is overloaded, then there are number of things you can do to optimise that part of your server.
tuning the mysql parameters.
optimising your database structure with faster indexes
organising data in a different way, or re-engineering you application to make better queries.
Focusing on how you application talks to your database (via a internal Unix domain sockets, or via a loopback TCP socket, does it reused connections pools)
'The best query to database , is one that you never have to make'. You think the 'youtube video number of views counter' is calculated with a database query? think again!
there are many tools to do, this some pay for tools/services , some you can just install for free.
https://github.com/major/MySQLTuner-perl
https://stackify.com/mysql-tutorial-improve-performance/
If it turns out that it's just the 'way' you accessing the data within the database, then think about you data? are you doing complex queries, or is mysql essential doing key/value pair looks ups... Experiment with different database servers , and take a look at nosql databases , your might have a database model that just fits these much faster ways of access data :-).
If you cannot rearchitect your database layer, then there are number of cache layers that can really speed up you application if it's making the same read queries over and over.
https://dev.mysql.com/doc/refman/5.7/en/innodb-memcached-intro.html
The last thing to think about is , who is accessing mysql? is the backend exposed to the internet? are all the request coming from where you think they should be? Do you have a backup job running remotely, that causes an already loaded server to become overloaded during the backup cycle?
Optimisation is a good think to learn, its ironic that many applications out there could support their current external loads with much less server hardware, and without scaling out. Scaling up is much easier (more NET/RAM/CPU/SSD/DISK).
Good luck.
https://www.tecmint.com/command-line-tools-to-monitor-linux-performance/
https://www.binarytides.com/linux-commands-monitor-network/
