Slow response from Identity Server 4.0 (asp.net core 2.0) on connect/token endpoint on high load

Question

We have setup Identity Server 4 with Entity Framework Core. We successfully created and Seeded the desired Client to the MSSQL Database and deployed both the Project and the Database to one and the same AWS Datacenter.

In our Startup.cs class we have:

           .AddInMemoryCaching()
          .AddClientStoreCache<CachingClientStore<ClientStore>>()
          .AddResourceStoreCache<CachingResourceStore<ResourceStore>>()
          .AddTestUsers(Config.GetUsers());

           where Config.GetUsers() is 

           public static List<TestUser> GetUsers()
          {
              return new List<TestUser>
              {
                  new TestUser
                  {
                      SubjectId = "1",
                      Username = "alice",
                      Password = "password"
                  },
                  new TestUser
                  {
                      SubjectId = "2",
                      Username = "bob",
                      Password = "password"
                  }
              };
          }

We successfully obtain an access_token with the provided configurations without making calls to the DB.

The Issue is as follow:

Using JMeter we face the following results:

2 Containers (1.5GB RAM) /connect/token (POST) endpoint (250 simultaneous requests issued for 1 second)

• Average response time – 252 ms

• Min response time – 142 ms

• Max response time – 597 ms

• Throughput – 167 req/sec

/connect/token (POST) endpoint (350 simultaneous requests issued for 1 second)

• Average response time – 780 ms

• Min response time – 150 ms

• Max response time – 2382 ms

• Throughput – 130 req/sec

/connect/token (POST) endpoint (400 simultaneous requests issued for 1 second)

• Average response time – 1112 ms

• Min response time – 153 ms

• Max response time – 3806 ms

• Throughput – 134 req/sec

/connect/token (POST) endpoint (500 simultaneous requests issued for 1 second)

• Average response time – 1456 ms

• Min response time – 153 ms

• Max response time – 3327 ms

• Throughput – 139 req/sec

/connect/token (POST) endpoint (750 simultaneous requests issued for 1 second)

• Average response time – 2379 ms

• Min response time – 349 ms

• Max response time – 4936 ms

• Throughput – 142 req/sec

/connect/token (POST) endpoint (1000 simultaneous requests issued for 1 second)

• Average response time – 5432 ms

• Min response time – 497 ms

• Max response time – 12501 ms

• Throughput – 82 req/sec

We noticed that the initial requests are processed extremely adequately, but on half of the test, the requests execution freezes, which reflects on the overall result.

Additionally, we have another service that executes similar operations, using the same containers and the same DB Type, an on the same test the results are:

2 Containers (512 RAM) (500 simultaneous requests issued for 1 second)

• Average response time – 145 ms

• Min response time – 124 ms

• Max response time – 1199 ms

• Throughput – 148 req/sec

(1000 simultaneous requests issued for 1 second)

• Average response time – 155 ms

• Min response time – 124 ms

• Max response time – 1450 ms

• Throughput – 211 req/sec

The case we have is that we need to cover 1000 simultaneous requests in 3 seconds interval or less. We have the extensibility to load balance containers and respectively increase the count of the containers, but the tests show that the results in the case of Identity Server 4 are not satisfying enough.

Are there any specific configurations we need to apply in order to achieve the above mentioned results.

Please advise, your contribution is highly appreciated and needed.

What's the CPU on the app server looking like during this? Since it's doing crypto signing it may well just be saturating the CPU. If you need more throughput then you'll probably just need more servers/cores available. — mackie, Feb 01 '18 at 09:42
We noticed the CPU gets exhausted with the first tests and that is why we used c5.2xlarge AWS machines (8 vCPU, 16GB Ram - CPU optimized machines) for the tests that I shared results above. With these machines the CPU utilization is not more than 45%. We are using 2 Ubuntu Docker containers - each containers is on a separate host. We noticed that the two slow operations are when calling ITokenRequestValidator -> ValidateRequestAsyncand IClientSecretValidator -> ValidateAsync — Mariyan Stefanov, Feb 01 '18 at 11:51

score 1 · Answer 1 · answered Feb 06 '18 at 13:15

1

Turned out that the most time consuming operation is generating the "refresh token". When we turned off that in the Clients (by removing offline_access), then the delay that was mentioned above disappeared. The Identity Server started to fly by increasing its throughput with 22.5 times in our case. Hope this information is going to help others.

answered Feb 06 '18 at 13:15

Mariyan Stefanov

289
3
13

I guess that's fine if you don't need refresh tokens. I'm curious as to why they're so expensive to generate though. – mackie Feb 06 '18 at 14:01
Honestly said we did not researched further - once we found out that this is causing the delay we just stopped using these – Mariyan Stefanov Feb 21 '18 at 08:59

Slow response from Identity Server 4.0 (asp.net core 2.0) on connect/token endpoint on high load

1 Answers1