Azure Webjob HTTP 409 Conflict errors

Question

I had a C# WebJob that was working nicely with the alpha WebJob api. I just updated it to the beta release, and after fixing connection strings and namespaces I get a HTTP 409 (Conflict) error when the JobHost tries to connect.

Here is the call stack:

Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.Protocol.TableOperationHttpResponseParsers.TableOperationPreProcess(Microsoft.WindowsAzure.Storage.Table.TableResult result, Microsoft.WindowsAzure.Storage.Table.TableOperation operation, System.Net.HttpWebResponse resp, System.Exception ex)   Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.TableOperation.InsertImpl.AnonymousMethod__2(Microsoft.WindowsAzure.Storage.Core.Executor.RESTCommand<Microsoft.WindowsAzure.Storage.Table.TableResult> cmd, System.Net.HttpWebResponse resp, System.Exception ex, Microsoft.WindowsAzure.Storage.OperationContext ctx) Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync<Microsoft.WindowsAzure.Storage.Table.TableResult>(Microsoft.WindowsAzure.Storage.Core.Executor.RESTCommand<Microsoft.WindowsAzure.Storage.Table.TableResult> cmd, Microsoft.WindowsAzure.Storage.RetryPolicies.IRetryPolicy policy, Microsoft.WindowsAzure.Storage.OperationContext operationContext)  Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.TableOperation.Execute(Microsoft.WindowsAzure.Storage.Table.CloudTableClient client, Microsoft.WindowsAzure.Storage.Table.CloudTable table, Microsoft.WindowsAzure.Storage.Table.TableRequestOptions requestOptions, Microsoft.WindowsAzure.Storage.OperationContext operationContext)  Unknown
Microsoft.WindowsAzure.Storage.dll!Microsoft.WindowsAzure.Storage.Table.CloudTable.Execute(Microsoft.WindowsAzure.Storage.Table.TableOperation operation, Microsoft.WindowsAzure.Storage.Table.TableRequestOptions requestOptions, Microsoft.WindowsAzure.Storage.OperationContext operationContext)    Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.Host.Storage.SdkCloudStorageAccount.Table.GetOrInsert<Microsoft.Azure.Jobs.Host.Runners.HostEntity>(Microsoft.Azure.Jobs.Host.Runners.HostEntity entity) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.Host.Runners.HostTable.GetOrCreateHostId(string hostName)    Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHostContext.JobHostContext(string dashboardConnectionString, string storageConnectionString, string serviceBusConnectionString, Microsoft.Azure.Jobs.ITypeLocator typeLocator, Microsoft.Azure.Jobs.INameResolver nameResolver)   Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.GetHostContext(Microsoft.Azure.Jobs.ITypeLocator typesLocator, Microsoft.Azure.Jobs.INameResolver nameResolver)  Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost(System.IServiceProvider serviceProvider) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost(Microsoft.Azure.Jobs.JobHostConfiguration configuration) Unknown
Microsoft.Azure.Jobs.Host.dll!Microsoft.Azure.Jobs.JobHost.JobHost()    Unknown

Exception text:

[Microsoft.WindowsAzure.Storage.StorageException] {"The remote server returned an error: (409) Conflict."} Microsoft.WindowsAzure.Storage.StorageException

packages.config:

package id="Microsoft.Azure.Jobs" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Azure.Jobs.Core" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Azure.Jobs.ServiceBus" version="0.3.1-beta" targetFramework="net45" package id="Microsoft.Bcl" version="1.1.7" targetFramework="net45" package id="Microsoft.Bcl.Build" version="1.0.14" targetFramework="net45" package id="Microsoft.Data.Edm" version="5.6.0" targetFramework="net45"
package id="Microsoft.Data.OData" version="5.6.0" targetFramework="net45" package id="Microsoft.Data.Services.Client" version="5.6.0" targetFramework="net45" package id="Microsoft.Net.Http" version="2.2.19" targetFramework="net45"
package id="Microsoft.WindowsAzure.ConfigurationManager" version="2.0.3" targetFramework="net45" package id="Newtonsoft.Json" version="6.0.4" targetFramework="net45" package id="System.Spatial" version="5.6.0" targetFramework="net45" package id="WindowsAzure.MobileServices" version="1.3.0-alpha4" targetFramework="net45" package id="WindowsAzure.ServiceBus" version="2.4.2.0" targetFramework="net45" package id="WindowsAzure.Storage" version="4.2.0" targetFramework="net45"

Any ideas?

Can you please give more info? Does it happen all the time? Do you have a stacktrace? — Victor Hurdugaci, Aug 06 '14 at 20:31
@VictorHurdugaci I've also tried the new JobHostConfiguration...no change. Do I need to do anything in the Azure portal to configure the storage? — tillerstarr, Aug 06 '14 at 22:40
Can you please try to delete the AzureJobsHosts table from your storage account? — Victor Hurdugaci, Aug 06 '14 at 23:18
@VictorHurdugaci I've deleted all the "Azure.." containers. Is that what you mean? I've also added the packages.config entries. Btw... no change. — tillerstarr, Aug 06 '14 at 23:43
@VictorHurdugaci After deleting the AzureJobHost (in VS) I'm able to "continue" past the exception. Is this expected behavior? — tillerstarr, Aug 07 '14 at 00:15
No, that's not expected. I don't know why you were getting the 409... I'll try to investigate on my side. If you somehow manage to reproduce the issue, please let me know — Victor Hurdugaci, Aug 07 '14 at 16:18
I am getting the same exception. I am simply running a sample WebJob with logging to same blob. I start the WebJob Once there are many queued items. — Joy George Kunjikkuru, Jul 14 '15 at 18:48

score 4 · Answer 1 · answered Sep 24 '15 at 15:24

Changing the jobRecurrenceFrequency in webjob-publish-settings.json to "Hour" worked for my https://error404.atomseo.com project!

I had the same problem and it turned out that publish process failed because I set it to recur every 10 minutes while the app was meant to run in free tier. As MS describes here:

https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-deploy-webjobs/

one can deploy with all frequencies other than those defined in minutes.

score 2 · Answer 2 · answered Jul 14 '15 at 19:19

I did some more research and could see that this can happen if there are concurrent requests to create queue/table with same name. Azure seems rejecting those requests with custom 409 http code which is meant for applications.

To reproduce even in the latest Azure SDK, just have a webjob to process queues and start the JobHost only after having many requests in queue. In the Job function Get the blob as TextWriter and write into it. Makes sure the blob 'logs/webjob1' is not present when the JobHost starts to see the issue.

public static void ProcessQueueMessage([QueueTrigger("testqueue1")] string inputText,
       [Blob("logs/webjob1")]TextWriter writer)
        {
            writer.WriteLine(inputText);
        }

I just had 4 items in the queue when I started JobHost and was able to see the issue from Azure. Used free hosting.

score 1 · Answer 3 · answered Sep 26 '18 at 11:15

I was getting the same error code

2018-08-18T23:06:02.2102822Z ##[error]Conflict
2018-08-18T23:06:02.2119417Z ##[error]Unable to retrieve connection details for Azure App Service : KaktusWatch. Status Code: 409 (Conflict)
2018-08-18T23:06:02.2150200Z ##[section]Finishing: Deploy Azure App Service

In my case the webjob-publish-settings.json was obsolete.

I had to fill in interval and jobRecurrenceFrequency settings parameters.

score 1 · Answer 4 · answered May 15 '21 at 13:22

We started receiving a 409 Conflict exception from Azure Web Jobs shortly after a release thrown from Microsoft.WindowsAzure.Storage. Not long before this release we had implemented archiving of older blobs. It turns out having readonly blobs in our storage (Archived) was the reason behind this exception being thrown in our case. It was not immediately evident that this was the culprit, I suspect it took time before the webjob started hitting the archived blobs. Once we removed the archived messages from our blob storage, the webjob was able to resume.

score 0 · Answer 5 · answered Apr 01 '22 at 17:50

0

When I was getting this error, I realized that the Jenkins job I was calling didn't have the parameters I was passing to it. Once I added the parameters, it worked fine.

answered Apr 01 '22 at 17:50

twasbrillig

17,084
9
43
67

Azure Webjob HTTP 409 Conflict errors

5 Answers5