I have:
- An express application which exposes a REST api and needs to connect to a MySQL database (see below)
- An AWS RDS MySQL 5.6.34 database on a
db.t2.medium
instance
In the case of a single request or "low" traffic there are no problems, but
if I try to stress the system sending an higher load of requests (I am using artillery for http load testing) I notice that basically all the requests hang until they fail with ECONNRESET
or ESOCKETTIMEDOUT
errors.
In the following test I am sending 30 GET requests/second for 10 seconds:
config:
target: 'http://localhost:8080'
phases:
- duration: 10
arrivalRate: 30
scenarios:
- flow:
- log: "New virtual user running"
- get:
url: '/v1/myresource'
headers:
apikey: 'myapikey'
This is the result:
All virtual users finished
Summary report @ 17:06:35(+0100) 2021-03-21
Scenarios launched: 300
Scenarios completed: 0
Requests completed: 0
Mean response/sec: 2.31
Response time (msec):
min: NaN
max: NaN
median: NaN
p95: NaN
p99: NaN
Scenario counts:
0: 300 (100%)
Errors:
ESOCKETTIMEDOUT: 251
ECONNRESET: 49
As you can see all the requests fail.
On the other hand, if I export the whole database to a local MySQL running on my laptop and try the same test, all works properly:
All virtual users finished
Summary report @ 17:08:14(+0100) 2021-03-21
Scenarios launched: 300
Scenarios completed: 300
Requests completed: 300
Mean response/sec: 18.79
Response time (msec):
min: 3518.9
max: 9439
median: 7795.5
p95: 9237.9
p99: 9281.9
Scenario counts:
0: 300 (100%)
Codes:
200: 300
I have considered a capacity problem on the RDS instance, but db metrics seem to indicate otherwise:
I don't think is a deadlock problem either, since these GET requests involve only simple SELECT statements on the database (no writes).
Does anyone have any suggestion on how I can troubleshoot this issue?
As additional informations, I am using knex.js as query builder library and mysql2 as the driver library. The following is the connection configuration:
const knex = require('knex')({
debug: true,
client: 'mysql2',
connection: {
host: 'localhost',
port: 3306,
user: 'myuser',
password: 'mypwd',
database: 'mydb',
charset: 'utf8mb4_bin'
},
pool: {
min: 1,
max: 5,
}
});