1

Are there any time limits on the queue triggered function inside an azure web job. My function takes about 20-30 mins since it needs to go over a lot of records in the DB. The function status in the end is Never Finished. I put in the settings.job file with { "stopping_wait_time": 1800 } - with no effect. But within five minutes or so its status is set to Never Finished. I searched and yes found some ppl asking similar questions - with no definitive answers. I found some exceptions in the logs and the exceptions seemed to be generated from Azure storage. I searched and found there were some issues with logging, so i commented out logging code within the function - still no go. Also i found one question that mentioned some connection strings - so now i have these; AzureWebJobsStorage, AzureWebJobsDashboard, AzureJobsRuntime, AzureJobsData - all pointing to the same storage account - still no go.

I tried to debug, and see where the timeout occurs and it seems like it is happening even before my function is actually hit,

Output is something like below;

A first chance exception of type 'System.Net.WebException' occurred in System.dll
A first chance exception of type 'System.Net.WebException' occurred in System.dll
A first chance exception of type 'System.TimeoutException' occurred in Microsoft.WindowsAzure.Storage.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in Microsoft.WindowsAzure.Storage.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in Microsoft.WindowsAzure.Storage.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in mscorlib.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in mscorlib.dll
A first chance exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in Microsoft.Azure.WebJobs.Host.dll
System.Transactions Critical: 0 : <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical"><TraceIdentifier>http://msdn.microsoft.com/TraceCodes/System/ActivityTracing/2004/07/Reliability/Exception/Unhandled</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>PnCProcessor.exe</AppDomain><Exception><ExceptionType>Microsoft.WindowsAzure.Storage.StorageException, Microsoft.WindowsAzure.Storage, Version=4.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35</ExceptionType><Message>The client could not finish the operation within specified timeout.</Message><StackTrace>   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndExists(IAsyncResult asyncResult)
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.&amp;lt;&amp;gt;c__DisplayClass1`1.&amp;lt;CreateCallback&amp;gt;b__0(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.&amp;lt;ExecuteAsync&amp;gt;d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.&amp;lt;RunAsync&amp;gt;d__d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.&amp;lt;&amp;gt;c__DisplayClass1.&amp;lt;Throw&amp;gt;b__0()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()</StackTrace><ExceptionString>Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---&amp;gt; System.TimeoutException: The client could not finish the operation within specified timeout.
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndExists(IAsyncResult asyncResult)
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.&amp;lt;&amp;gt;c__DisplayClass1`1.&amp;lt;CreateCallback&amp;gt;b__0(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.&amp;lt;ExecuteAsync&amp;gt;d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.&amp;lt;RunAsync&amp;gt;d__d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.&amp;lt;&amp;gt;c__DisplayClass1.&amp;lt;Throw&amp;gt;b__0()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
Request Information
RequestID:
RequestDate:
StatusMessage:
</ExceptionString><InnerException><ExceptionType>System.TimeoutException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>The client could not finish the operation within specified timeout.</Message><StackTrace>   at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.&amp;lt;&amp;gt;c__DisplayClass1.&amp;lt;Throw&amp;gt;b__0()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

System.TimeoutException: The client could not finish the operation within specified timeout.

I am using a S1 web app and a standard-GRS storage account for this.

So questions are;
1. Are there any time limitations on the time the queue triggered function takes?
2. If there are time limitations is there any way to override those?
3. I have to use a queue triggered function since i need to pass some parameters to the function - which are passed from another scheduled web job. I can make an on-demand webjob and pass the parameters on the command line - but then is there a way to call an on-demand webjob from a scheduled webjob?

Thanks for your help!

user5845859
  • 11
  • 1
  • 3
  • From the stack trace it looks like you're getting storage timeouts during startup initialization (e.g. checking if the queue even exists). That indicates a problem with your storage account or connection string. Make sure these are correct. Also, please provide the details on how you are running this job -triggered/continuous, etc. – mathewc Jan 27 '16 at 16:21
  • thanks for your response mathew, that was my hypothesis as well - however the same setup was working fine when the function had lesser number of records to process and the function finished in a lot lesser time. One more thing that i suspect might relate to this is that the same storage account had the queue which created these records in the first place, like around 500,000 messages. So there are lots and lots of files in the blobs container - does that matter? and is there anyway to remove those? – user5845859 Jan 27 '16 at 18:55
  • ..and the job is triggered via a queue – user5845859 Jan 27 '16 at 18:56
  • ok, i deleted the old storage, created a new one. Changed the AzureWebJobsStorage, AzureWebJobsDashboard to point to the new one. i changed the function triggered by the queue to just spit out one msg in the TextWriter and return. All that worked. So the queue-storage-log functionality is working. However if i change the function again to do the long running job, again it gets killed. So the same question again. Is there a time limit for the function that gets executed by a message off of a queue? – user5845859 Jan 29 '16 at 14:59

1 Answers1

1

No there is no limit on execution time of queue triggered job functions. However there are a few things you have to ensure you set up properly so your JobHost continues to run. When using the Azure WebJobs SDK you need to deploy your JobHost Console exe as a continuously running WebJob, and enable Always On. Both of those are described in detail on this documentation page under the "Create a continuously running WebJob" section.

See the "Notes" section of that page for relevant details on the timeout limits when running in the Free tier (20 minute timeout), which can be removed by moving to Basic or Standard tier.

mathewc
  • 13,312
  • 2
  • 45
  • 53