Background
Our API allows users to upload large files, like PDFs, JPGs and PNGs. Recently we have experienced a ton of requests that times out when the server has been idle for some time. Our clients would experience 2 minutes of wait time and then receive a 500 error message.
We use Azure API Management to provide clients with documentation and access to the APIs.
Client -> API Management -> Web App
However, after looking at insight it is clear that the issue is between API Management and our Web App, which is where we have a SSL certificate set up.
The issue
The issue seems to be that sending a large POST request containing a SSL certificate to "wake" the API makes it deadlock. I've debugged the API using application insight as well, but it seems like the request is not received by any operation at all.
After debugging the issue with a brand new Web API project and Web App, I have narrowed it down to that the timeout only occurs if:
- The server is configured for HTTPS only and/or Client Certificate Required.
- I let the server be for 5-10 minutes before sending my first request.
- My first request contains a large file, like a 2 MB PDF file.
I set up the following resources in order to debug the issue.
Web API
I created a brand new project, ASP.NET Web API, framework 4.6. I only added a simple files controller.
using System.Net;
using System.Net.Http;
using System.Web.Http;
namespace DebugCertificate.API.Controllers
{
public class FilesController : ApiController
{
public HttpResponseMessage Post()
{
// Success
return Request.CreateResponse(HttpStatusCode.OK, "Welcome to the POST files resource!");
}
public HttpResponseMessage Get()
{
// Success
return Request.CreateResponse(HttpStatusCode.OK, "Welcome to the GET files resource!");
}
}
}
Web App
I set up a brand new Azure Web App and configured it with the following settings.
Steps to reproduce the issue with Postman:
- Wait 5-10 minutes.
- Send a POST request to the /api/files resource containing a large file (2 MB in this case).
- The timeout occur after 120 seconds.
- As long as you only send large POST request, you can repeat step 2.
Notice that I don't even have to send a client certificate with this request for the issue to occur.
The issue will not occur
- When require client certificate and HTTPS only is off.
- When you send a GET /api/files request first and within 5-10 minutes send the same POST request described in step 2 above.
Possible leads
After diagnosing the Web App in Azure I found the errors that are generated by these requests. I have Googled this description but have not found any information that would fix my issues.
I found something that sounded similar, but this article sounds like it is between the client and API Management, which is not where we have the SSL certificate set up. Look for the green TIP box.
How to secure APIs using client certificate authentication in API Management
I also found this article containing a few ways to solve the issue, but "priming" the API with an additional request seems like a bit of a hassle.
HTTPS Client Certificate Request freezes when the Server is handling a large PUT/POST Request
Questions
- Why is this happening?
- Is there any workaround that does not involve "priming" the API? A simple Web.config setting maybe?
Let me now if more information is required!
