1

I have webjobs deployed in Azure and it was previously working before but after our latest deployment we are experiencing issues that our webjobs are throwing "Azure Storage Exception".

  [03/25/2016 01:38:26 > 5f45b8: SYS INFO] Status changed to Initializing
    [03/25/2016 01:38:40 > 5f45b8: SYS INFO] Run script 'ListNet.JobController.exe' with script host - 'WindowsScriptHost'
    [03/25/2016 01:38:40 > 5f45b8: SYS INFO] Status changed to Running
    [03/25/2016 01:38:41 > 5f45b8: INFO] Found the following functions:
    [03/25/2016 01:38:41 > 5f45b8: INFO] ListNet.JobController.Functions.ManualTrigger
    [03/25/2016 01:38:41 > 5f45b8: INFO] Executing: 'Functions.ManualTrigger' because This was function was programmatically called via the host APIs.
    [03/25/2016 01:41:12 > 5f45b8: ERR ] 
    [03/25/2016 01:41:12 > 5f45b8: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.WindowsAzure.Storage.Blob.CloudBlobSharedImpl.<DeleteBlobImpl>b__1b(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    --- End of inner exception stack trace ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndDelete(IAsyncResult asyncResult)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar)
    [03/25/2016 01:41:12 > 5f45b8: ERR ] --- End of stack trace from previous location where exception was thrown ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.Azure.WebJobs.Host.Protocols.PersistentQueueWriter`1.<DeleteAsync>d__6.MoveNext()
    [03/25/2016 01:41:12 > 5f45b8: ERR ] --- End of stack trace from previous location where exception was thrown ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.Azure.WebJobs.Host.Loggers.CompositeFunctionInstanceLogger.<DeleteLogFunctionStartedAsync>d__e.MoveNext()
    [03/25/2016 01:41:12 > 5f45b8: ERR ] --- End of stack trace from previous location where exception was thrown ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()
    [03/25/2016 01:41:12 > 5f45b8: ERR ] --- End of stack trace from previous location where exception was thrown ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.Azure.WebJobs.Host.Executors.HeartbeatFunctionExecutor.<TryExecuteAsync>d__0.MoveNext()
    [03/25/2016 01:41:12 > 5f45b8: ERR ] --- End of stack trace from previous location where exception was thrown ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.Azure.WebJobs.Host.Executors.AbortListenerFunctionExecutor.<TryExecuteAsync>d__0.MoveNext()
    [03/25/2016 01:41:12 > 5f45b8: ERR ] --- End of stack trace from previous location where exception was thrown ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.Azure.WebJobs.Host.Executors.ShutdownFunctionExecutor.<TryExecuteAsync>d__0.MoveNext()
    [03/25/2016 01:41:12 > 5f45b8: ERR ] --- End of stack trace from previous location where exception was thrown ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.Azure.WebJobs.JobHost.<CallAsyncCore>d__7.MoveNext()
    [03/25/2016 01:41:12 > 5f45b8: ERR ] --- End of stack trace from previous location where exception was thrown ---
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at Microsoft.Azure.WebJobs.JobHost.Call(MethodInfo method)
    [03/25/2016 01:41:12 > 5f45b8: ERR ]    at ListNet.JobController.Program.Main()
    [03/25/2016 01:41:12 > 5f45b8: SYS INFO] Status changed to Failed
    [03/25/2016 01:41:12 > 5f45b8: SYS ERR ] Job failed due to exit code -532462766

I looked into this similar issue but I already have these containers on my azure storage:

Azure Web Job-The remote server returned 404

https://i.stack.imgur.com/HkCab.png

Notes

  • I use manual trigger
  • I create a default web job and run it successfully
Community
  • 1
  • 1
Kimwan Ogot
  • 456
  • 4
  • 11
  • 1
    The "AzureWebJobsStorage" storage connection string is correct and not being overridden by something unexpected in the WebApp's portal application settings? – Bryan Lewis Mar 25 '16 at 06:29
  • AzureStorageConnection string is correct. The connection string on my AzureWebJob Configuration is the same as my connection string on the app.config should I remove this connection string from Configure tab of azure web job? – Kimwan Ogot Mar 25 '16 at 06:50
  • It now works when I remove the connection string in the config. – Kimwan Ogot Mar 25 '16 at 08:19
  • The portal wepapp settings should override the web.config/app.config... – Bryan Lewis Mar 25 '16 at 08:22
  • is it safe to remove it since I don't have a web app deployed. We only use the web app to manage web jobs – Kimwan Ogot Mar 25 '16 at 12:26
  • I have **AzureWebJobsDashboard** connection string on my web app's web config and removing this will make my webjob succeed but I will not have a reference to the logs that shows what function did the webjob triggers – Kimwan Ogot Mar 25 '16 at 14:46
  • Does your dashboard point to the same storage account? – Bryan Lewis Mar 25 '16 at 16:59
  • Yes, my dashboard point to the same storage account – Kimwan Ogot Mar 26 '16 at 04:06
  • Something is fishy. As per your log the web job failed **after** it triggered the function __ManualTrigger__. Generally if the web job can't connect to the storage account it fails to even **before** executing any function Can you please show some code? – Sandesh Mar 30 '16 at 10:44
  • That is correct @Sandesh. This is really annoying, what code would you want to see? – Kimwan Ogot Apr 01 '16 at 16:05
  • `public class Program { // Please set the following connection strings in app.config for this WebJob to run: // AzureWebJobsDashboard and AzureWebJobsStorage private static void Main() { var host = new JobHost(); // The following code ensures that the WebJob will be running continuously //host.RunAndBlock(); host.Call(typeof (Functions).GetMethod("ManualTrigger")); } }` – Kimwan Ogot Apr 01 '16 at 16:11
  • `public static void ManualTrigger(TextWriter log) { var master = SaveMasterFileToStorage(); log.WriteLine("End of saving master, total is {0}", master.MasterPagination.TotalRecords); DataProcessingService.RemoveExpiredListings(master); log.WriteLine("End of removing expired listings"); log.WriteLine("Done!"); }` – Kimwan Ogot Apr 01 '16 at 16:14
  • Can you please put this code in the question itself so that it is better readable. Also one small suggestion. Can you wrap your code inside ManualTrigger function inside try/catch block and see if you get the same exception. Instead of using ***log*** please use ***Console.WriteLine*** – Sandesh Apr 02 '16 at 06:35

1 Answers1

1

After months of verification I have found out that the cause of this issue is duplicate nuget package dependencies. It turns out that I have multiple nuget package (Newtonsoft.Json) with different versions (e.g: 1.1, 1.2).

On why nuget is doing that, I have no idea.

Kimwan Ogot
  • 456
  • 4
  • 11
  • I was getting this same error. What worked for me was updating the `WindowsAzure.Storage` package in my case from `4.3.0` to `7.1.2`, but I would go with latest or whatever your other WebJobs are using (if they are running without issue). – Michael Johnston Oct 18 '16 at 23:12