Express-Gateway - Basic Auth / Redis Performances Issue

Question

We started using express-gateway as a simple proxy with some rewrite and jwt verification (jwt comes from Auth0). No problem there.

For some endpoints I needed basic-auth so I needed to set up persistence.

We use Kubernetes to run our apps and managed databases all on the same cloud provider.

I noticed long response time for endpoints using basic-auth. I started running some basic benchmarks. Here are the results.

Results Summary

Directly Calling my App
- 284 Requests/sec
Express Gateway - Proxy Only
- 285 Requests/sec
Express Gateway - Proxy + Logs
- 270 Requests/sec
Express Gateway (Managed Redis) - Proxy + Logs + Basic Auth
- 7 Requests/sec
Express Gateway (Embedded Redis) - Proxy + Logs + Basic Auth
- 7 Requests/sec

I first though that my managed Redis had poor performances (its supposed to be on the same data-center has the kubernetes cluster) but the embedded Redis has the same performances.

I don't need huge performances but it seams to me that there is an issue with the Basic Auth / Redis implementation.

Is this expected ? Do I need a huge Redis database ? What can I do to troubleshoot this issue ?

Results Details

Shared

HTTP benchmarks are done with HEY.

Express-Gateway docker image : express-gateway:1.16.x

Express-Gateway gateway.config.yml (redacted) :

http:
  port: ${HTTP_PORT:-8080}
admin:
  port: ${ADMIN_PORT:-9876}
  host: ${ADMIN_HOSTNAME:-localhost}
apiEndpoints:
  app-api:
    - host: ${APP_HOST}
      paths:
        - '/some/path'
      methods: ["POST"]
      scopes: ["scope:write", "scope:read"]
serviceEndpoints:
  app-api:
    url: ${SERVICE_HOST_APP}
policies:
  - log
  - proxy
  - basic-auth
pipelines:
  app-api:
    apiEndpoints:
      - app-api
    policies:
      - log: 
        - action:
          message: "${req.method} ${req.originalUrl}"
      - basic-auth:
      - proxy:
        - action:
            changeOrigin: false
            serviceEndpoint: app-api

Express-Gateway Resources :

resources:
  requests:
    memory: "512M"
    cpu: "500m"
  limits:
    memory: "1024M"
    cpu: "1000m"

App Resources :

resources:
  requests:
    memory: "512M"
    cpu: "500m"
  limits:
    memory: "1024M"
    cpu: "1000m"

1. Directly Calling my App

Command :

./hey_linux_amd64 -n 10000 -m POST -T "application/json" -d <body> <app url>

Results :

Summary:
  Total:    35.2033 secs
  Slowest:  0.6221 secs
  Fastest:  0.0335 secs
  Average:  0.1685 secs
  Requests/sec: 284.0639


Response time histogram:
  0.034 [1] |
  0.092 [2689]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.151 [1830]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.210 [1781]  |■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.269 [2197]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.328 [1271]  |■■■■■■■■■■■■■■■■■■■
  0.387 [130]   |■■
  0.446 [31]    |
  0.504 [37]    |■
  0.563 [22]    |
  0.622 [11]    |


Latency distribution:
  10% in 0.0435 secs
  25% in 0.0852 secs
  50% in 0.1643 secs
  75% in 0.2434 secs
  90% in 0.2834 secs
  95% in 0.3056 secs
  99% in 0.3944 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0007 secs, 0.0335 secs, 0.6221 secs
  DNS-lookup:   0.0001 secs, 0.0000 secs, 0.0316 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0083 secs
  resp wait:    0.1676 secs, 0.0334 secs, 0.6217 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0027 secs

Status code distribution:
  [200] 10000 responses

Resource Usage :

> kubectl top pods
NAME                                       CPU(cores)   MEMORY(bytes)   
app-6fb8b4787d-9qpf2                       920m         254Mi

2. Express Gateway - Proxy Only

Command :

./hey_linux_amd64 -m POST -T "application/json" -d <body> <express-gateway url>

Results:

Summary:
  Total:    35.0445 secs
  Slowest:  0.7521 secs
  Fastest:  0.0349 secs
  Average:  0.1672 secs
  Requests/sec: 285.3516


Response time histogram:
  0.035 [1] |
  0.107 [3145]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.178 [2298]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.250 [2286]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.322 [2046]  |■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.394 [153]   |■■
  0.465 [32]    |
  0.537 [25]    |
  0.609 [9] |
  0.680 [4] |
  0.752 [1] |


Latency distribution:
  10% in 0.0450 secs
  25% in 0.0865 secs
  50% in 0.1628 secs
  75% in 0.2436 secs
  90% in 0.2796 secs
  95% in 0.2989 secs
  99% in 0.3538 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0009 secs, 0.0349 secs, 0.7521 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0074 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0042 secs
  resp wait:    0.1662 secs, 0.0348 secs, 0.7520 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0028 secs

Status code distribution:
  [200] 10000 responses

Resource Usage :

> kubectl top pods
NAME                                       CPU(cores)   MEMORY(bytes)   
express-gateway-56699967d8-hnvzw           320m         50Mi
app-6fb8b4787d-9qpf2                       743m         252Mi

3. Express Gateway - Proxy + Logs

Command :

./hey_linux_amd64 -m POST -T "application/json" -d <body> <express-gateway url>

Results:

Summary:
  Total:    36.9277 secs
  Slowest:  0.6278 secs
  Fastest:  0.0356 secs
  Average:  0.1796 secs
  Requests/sec: 270.7991


Response time histogram:
  0.036 [1] |
  0.095 [1595]  |■■■■■■■■■■■■■■■■■■■■■■■■
  0.154 [2616]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.213 [2604]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.272 [1756]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.332 [996]   |■■■■■■■■■■■■■■■
  0.391 [301]   |■■■■■
  0.450 [71]    |■
  0.509 [22]    |
  0.569 [32]    |
  0.628 [6] |


Latency distribution:
  10% in 0.0734 secs
  25% in 0.1195 secs
  50% in 0.1709 secs
  75% in 0.2316 secs
  90% in 0.2932 secs
  95% in 0.3262 secs
  99% in 0.4061 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0007 secs, 0.0356 secs, 0.6278 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0038 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0053 secs
  resp wait:    0.1787 secs, 0.0355 secs, 0.6275 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0063 secs

Status code distribution:
  [200] 10000 responses

Resource Usage :

> kubectl top pods
NAME                                       CPU(cores)   MEMORY(bytes)   
express-gateway-56699967d8-hnvzw           760m         94Mi 
app-6fb8b4787d-9qpf2                       898m         253Mi

4. Express Gateway (Managed Redis) - Proxy + Logs + Basic Auth

Command :

./hey_linux_amd64 -m POST -H "Authorization: Basic ..." -T "application/json" -d <body> <express-gateway url>

Results:

Summary:
  Total:    1386.7479 secs
  Slowest:  11.4025 secs
  Fastest:  0.2489 secs
  Average:  6.9121 secs
  Requests/sec: 7.2111


Response time histogram:
  0.249 [1] |
  1.364 [15]    |
  2.480 [77]    |■
  3.595 [148]   |■
  4.710 [236]   |■■
  5.826 [493]   |■■■■
  6.941 [2836]  |■■■■■■■■■■■■■■■■■■■■■
  8.056 [5292]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  9.172 [595]   |■■■■
  10.287 [231]  |■■
  11.403 [76]   |■


Latency distribution:
  10% in 5.8535 secs
  25% in 6.8129 secs
  50% in 7.0067 secs
  75% in 7.1767 secs
  90% in 7.8473 secs
  95% in 8.5065 secs
  99% in 10.0107 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0008 secs, 0.2489 secs, 11.4025 secs
  DNS-lookup:   0.0001 secs, 0.0000 secs, 0.0252 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0072 secs
  resp wait:    6.9111 secs, 0.2488 secs, 11.4024 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0090 secs

Status code distribution:
  [200] 9990 responses
  [502] 10 responses

Resource Usage :

> kubectl top pods
NAME                                       CPU(cores)   MEMORY(bytes)   
express-gateway-56699967d8-hnvzw           966m         62Mi
app-6fb8b4787d-9qpf2                       20m          245Mi

5. Express Gateway (Embedded Redis) - Proxy + Logs + Basic Auth

Command :

./hey_linux_amd64 -m POST -H "Authorization: Basic ..." -T "application/json" -d <body> <express-gateway url>

Results:

Summary:
  Total:    1395.9956 secs
  Slowest:  11.2510 secs
  Fastest:  0.2258 secs
  Average:  6.9731 secs
  Requests/sec: 7.1633


Response time histogram:
  0.226 [1] |
  1.328 [3] |
  2.431 [13]    |
  3.533 [3] |
  4.636 [12]    |
  5.738 [145]   |■
  6.841 [2597]  |■■■■■■■■■■■■■■■
  7.943 [6804]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  9.046 [375]   |■■
  10.148 [29]   |
  11.251 [18]   |


Latency distribution:
  10% in 6.3767 secs
  25% in 6.8170 secs
  50% in 6.9824 secs
  75% in 7.1205 secs
  90% in 7.5867 secs
  95% in 7.8832 secs
  99% in 8.5095 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0007 secs, 0.2258 secs, 11.2510 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0093 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0053 secs
  resp wait:    6.9722 secs, 0.2255 secs, 11.2504 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0126 secs

Status code distribution:
  [200] 9993 responses
  [502] 7 responses

Resource Usage :

> kubectl top pods
NAME                                       CPU(cores)   MEMORY(bytes)  
express-gateway-56699967d8-hnvzw           651m         57Mi
app-6fb8b4787d-9qpf2                       2m           252Mi

The linked github issue https://github.com/ExpressGateway/express-gateway/issues/968 — Yannholo, Dec 09 '19 at 12:35

score 0 · Answer 1 · answered Jan 04 '20 at 10:55

I've replicated the same setup that you've posted here as much as I could and I'm not getting the same poor performances you're experiencing.

I created an Azure Redis Cache on the Cloud and running EG locally on my machine (MacBook Pro 2019) and here are the results:

Concurrency Level:      50
Time taken for tests:   3.425 seconds
Complete requests:      500
Failed requests:        0
Non-2xx responses:      500
Total transferred:      99500 bytes
HTML transferred:       6000 bytes
Requests per second:    145.99 [#/sec] (mean)
Time per request:       342.478 [ms] (mean)
Time per request:       6.850 [ms] (mean, across all concurrent requests)
Transfer rate:          28.37 [Kbytes/sec] received

I've been using the same gateway configuration. I would say there's something going on in your Kubernetes Networking configuration.

Express-Gateway - Basic Auth / Redis Performances Issue

Results Summary

Results Details

Shared

1. Directly Calling my App

2. Express Gateway - Proxy Only

3. Express Gateway - Proxy + Logs

4. Express Gateway (Managed Redis) - Proxy + Logs + Basic Auth

5. Express Gateway (Embedded Redis) - Proxy + Logs + Basic Auth

1 Answers1