0

I have:

  • An express application which exposes a REST api and needs to connect to a MySQL database (see below)
  • An AWS RDS MySQL 5.6.34 database on a db.t2.medium instance

In the case of a single request or "low" traffic there are no problems, but if I try to stress the system sending an higher load of requests (I am using artillery for http load testing) I notice that basically all the requests hang until they fail with ECONNRESET or ESOCKETTIMEDOUT errors.

In the following test I am sending 30 GET requests/second for 10 seconds:

config:
  target: 'http://localhost:8080'
  phases:
    - duration: 10
      arrivalRate: 30
scenarios:
  - flow:
      - log: "New virtual user running"
      - get:
          url: '/v1/myresource'
          headers:
            apikey: 'myapikey'

This is the result:

All virtual users finished
Summary report @ 17:06:35(+0100) 2021-03-21
  Scenarios launched:  300
  Scenarios completed: 0
  Requests completed:  0
  Mean response/sec: 2.31
  Response time (msec):
    min: NaN
    max: NaN
    median: NaN
    p95: NaN
    p99: NaN
  Scenario counts:
    0: 300 (100%)
  Errors:
    ESOCKETTIMEDOUT: 251
    ECONNRESET: 49

As you can see all the requests fail.
On the other hand, if I export the whole database to a local MySQL running on my laptop and try the same test, all works properly:

All virtual users finished
Summary report @ 17:08:14(+0100) 2021-03-21
  Scenarios launched:  300
  Scenarios completed: 300
  Requests completed:  300
  Mean response/sec: 18.79
  Response time (msec):
    min: 3518.9
    max: 9439
    median: 7795.5
    p95: 9237.9
    p99: 9281.9
  Scenario counts:
    0: 300 (100%)
  Codes:
    200: 300

I have considered a capacity problem on the RDS instance, but db metrics seem to indicate otherwise:

enter image description here enter image description here enter image description here

I don't think is a deadlock problem either, since these GET requests involve only simple SELECT statements on the database (no writes).

Does anyone have any suggestion on how I can troubleshoot this issue?

As additional informations, I am using knex.js as query builder library and mysql2 as the driver library. The following is the connection configuration:

const knex = require('knex')({
    debug: true,
    client: 'mysql2',
    connection: {
        host: 'localhost',
        port: 3306,
        user: 'myuser',
        password: 'mypwd',
        database: 'mydb',
        charset: 'utf8mb4_bin'
    },
    pool: {
        min: 1,
        max: 5,
    }
});
revy
  • 3,945
  • 7
  • 40
  • 85

1 Answers1

0

you can debug this issue by enabling the enhanced monitoring and check for CPU% and MEM% metrics. Please check below article on how to do it.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring.OS.html

Mohit Sahu
  • 177
  • 2
  • 17