2

The issue is with a long running azure webjob on a daily schedule. Each run takes 2-4 hours doing data analytics. The only dependencies are with Azure SQL database via EF and with Azure Storage, just setting up AzureWebJobsDashboard and AzureWebJobsStorage connections on the App.Config, standard setup on VS with webjob SDK. Most of the time of the webjob is consumed with EF's SaveChanges(). I also do an important amount of logging to monitor progress with aprox 3000 lines of Console output. The web app is configured as Allways ON and the WEBJOBS_IDLE_TIMEOUT is set to a very high number.

The following is the log of the error:

[10/20/2016 07:48:17 > 492c46: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.

[10/20/2016 07:48:17 > 492c46: ERR ] --- End of inner exception stack trace ---

[10/20/2016 07:48:17 > 492c46: ERR ] at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End()

[10/20/2016 07:48:17 > 492c46: ERR ] at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult)

[10/20/2016 07:48:17 > 492c46: ERR ] at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.b__3(IAsyncResult ar)

donquijote
  • 1,642
  • 5
  • 19
  • 41

2 Answers2

1

This is a few months old, but for those that come later...

You mention you do a lot of logging. There is an issue that was logged dealing with something similar. Apparently the WebJobs SDK does a periodic save of the log data to blob storage. If you are using a lot of bandwidth or otherwise consuming a lot of resources you may run into timeouts from the SDK trying to save to blob storage. Note the upload call in your stack trace.

I'm seeing this sporadically in a process that is punishing the wire pretty good so I am disabling logging via the WebJobs logging facility.

JasonCoder
  • 1,126
  • 2
  • 12
  • 24
0

A triggered webjob is aborted if it is idle, has no cpu time or output for a certain amount of time. Try and increase it by setting configuration WEBJOBS_IDLE_TIMEOUT to a large number, for instance 3600.

It could also be aborted if your instances hasn't configured Always on.

If that doesn't help you should try handle the amount of logging. Could it be that you try and write to many messages to fast? Have a look at this answer to see if that coould be the case.

Community
  • 1
  • 1
iikkoo
  • 2,810
  • 6
  • 33
  • 38
  • Thanks for pointing out, I have the WEBJOBS_IDLE_TIMEOUT configured to a large number and my instance is Always on. I will add these two points by editing my question above. Regarding your third suggestion my console output is around 3000 lines but I generate each of them at least one second apart. – donquijote Oct 21 '16 at 06:43
  • Have you had a look at https://github.com/Azure/azure-webjobs-sdk/issues/755? Try to downgrade your Azure Storage SDK. – iikkoo Oct 21 '16 at 13:31