0

I am doing performance testing of my Azure Web API that receives file attachments from the client and then uploads them to the Data Lake Store. My performance test is currently running for 6 minutes with a load of 250 users making 40 requests/sec.

The file uploads are successful until around 4minutes while the requests are under 4000, once the requests exceeds 4000 the file upload starts failing with the error of Port Exhaustion.

After some research I found out that there are around 4K ports available for communication and once the client sends the FIN packet, those ports go into a TcpTimedWaitDelay which by default is 4minutes(240seconds).

The solutions I found after initial research includes

1- Minimizing the TIME_WAIT of the ports by changing the registry.

My scenario: I'm using a Web API and I do not have access to the VM.

2- Increasing the ports to 65K by changing registry.

My scenario: I'm using a Web API and I do not have access to the VM.

3- Disposing the http client that is being used to make the requests.

My scenario: I do not have access to the client directly as I am using Azure .NET SDK's DataLakeStoreFileSystemManagementClient to upload the files.

I get the error after around 4K+ requests have been made. For file upload I use 

DataLakeStoreFileSystemManagementClient.FileSystem.Create(_dlAccountName, filePath, filestream, true)

Can someone please help fix this port exhaustion issue?

Sarmad
  • 303
  • 3
  • 16

2 Answers2

0

Something which jumps to mind is the session timeout on your file upload session. Once you hit the 4000 mark 6 minutes in then essentially you have no ports available until the earliest sessions start timing out and the transient client port connection resource on the server is released.

In a standard HTTP session environment you would have enormous flexibility to tune the session timeout to recover the ports in the configuration file for your web server/http-based applications server/HTTP ESB/ etc.... The timeout on your target seems to be set to 240 seconds. Do you have a configuration option available to reduce this value in the configuration of your target service?

James Pulley
  • 5,606
  • 1
  • 14
  • 14
  • Hi James, I am using `DataLakeStoreFileSystemManagementClient` from Azure SDK and it does not provide any timeout options on the client side. On the server side the only option I'm aware of is decreasing the Time_Wait value but unfortunately in the WebApi I do not have access to change the registry. – Sarmad Jan 17 '18 at 06:30
  • Looks like you have hit the tradeoff limit of SAAS in terms of flexibility on configuration – James Pulley Jan 17 '18 at 16:26
0

Actually there is way to update the default timeout of 5 minutes:

DataLakeStoreFileSystemClient.HttpClient.Timeout = TimeSpan.FromMinutes(1);

Also, please take note that we recently released a new Data Lake Store SDK just for filesystem operations in order to improve performance. Check it out!

Nuget: https://www.nuget.org/packages/Microsoft.Azure.DataLake.Store/

Github: https://github.com/Azure/azure-data-lake-store-net

Joo Wan Ro
  • 33
  • 4