32

I'm running an Azure function in Azure, the function gets triggered by a file being uploaded to blob storage container. The function detects the new blob (file) but then outputs the following message - Did not find any initialized language workers.

Setup:

  • Azure function using Python 3.6.8
  • Running on linux machine
  • Built and deployed using azure devops (for ci/cd capability)
  • Blob Trigger Function

I have run the code locally using the same blob storage container, the same configuration values and the local instance of the azure function works as expected.

The functions core purpose is to read in the .xml file uploaded into blob storage container and parse and transform the data in the xml to be stored as Json in cosmos db.

I expect the process to complete like on my local instance with my documents in cosmos db, but it looks like the function doesn't actually get to process anything due to the following error:

Did not find any initialized language workers

Cardstdani
  • 4,999
  • 3
  • 12
  • 31
nathan shumoogum
  • 353
  • 1
  • 4
  • 9
  • 1
    I'm wondering if it has anything to do with `FUNCTIONS_WORKER_RUNTIME` setting. Can you check in your `local.settings.json` file for this setting and settings in Azure Portal? Ref: https://learn.microsoft.com/en-us/azure/azure-functions/functions-run-local. – Gaurav Mantri Jul 01 '19 at 08:45
  • 2
    Yes I have set FUNCTIONS_WORKER_RUNTIME in both local.settings.json and in the Azure portal but still getting the error in the function in Azure portal (but not locally). – nathan shumoogum Jul 01 '19 at 13:27
  • 1
    I guess the language worker could have crashed. Try downloading the app content from the overview page and run it locally to see more detailed errors. – PramodValavala Jul 18 '19 at 12:08
  • 1
    @nathanshumoogum How did you get this issue resolved? – RB17 May 11 '20 at 13:10
  • @RB17 - Sorry for the slow reply but not working for for the company which this problem occurred for anymore and hence not working with Microsoft Azure. Hopefully some of the comments below will help you out – nathan shumoogum Jul 28 '20 at 14:49
  • I am having the same issue – Farzzy - MSFT Oct 18 '21 at 13:07

5 Answers5

5

Troy Witthoeft's answer was almost certainly the right one at the time the question was asked, but this error message is very general. I've had this error recently on runtime 3.0.14287.0. I saw the error on many attempted invocations over about 1 hour, but before and after that everything worked fine with no intervention.

I worked with an Azure support engineer who gave some pointers that could be generally useful:

  • Python versions: if you have function runtime version ~3 set under the Configuration blade, then the platform may choose any of python versions 3.6, 3.7, or 3.8 to run your code. So you should test your code against all three of these versions. Or, as per that link's suggestion, create the function app using the --runtime-version switch to specify a specific python version.

  • Consumption plans: this error may be related to a consumption-priced app having idled off and taking a little longer to warm back up again. This depends, of course, on the usage pattern of the app. (I infer (but the Engineer didn't say this) that perhaps if the Azure datacenter my app is in happens to be quite busy when my app wants to restart, it might just have to wait for some resources to become available.). You could address this either by paying for an always-on function app, or by rigging some kind of heartbeat process to stop the app idling for too long. (Easiest with a HTTP trigger: probably just ping it?)

  • The Engineer was able to see a lower-level error message generated by the Azure platform, that wasn't available to me in Application Insights: ARM authentication token validation failed. This was raised in Microsoft.Azure.WebJobs.Script.WebHost.Security.Authentication.ArmAuthenticationHandler.HandleAuthenticate() at /src/azure-functions-host/src/WebJobs.Script.WebHost/Security/Authentication/Arm/ArmAuthenticationHandler.cs. There was a long stack trace with innermost exception being: System.Security.Cryptography.CryptographicException : Padding is invalid and cannot be removed.. Neither of us were able to make complete sense of this and I'm not clear whether the responsibility for this error lies within the HandleAuthenticate() call, or outside (invalid input token from... where?).

The last of these points may be some obscure bug within the Azure Functions Host codebase, or some other platform problem, or totally misleading and unrelated.

Stuart J Cuthbertson
  • 390
  • 5
  • 13
  • May I ask... was your function on the consumption plan? How did you verify it's SDK version? Lastly, we've got three suggestions here... how did you resolve the _specific_ issue you were seeing? – Troy Witthoeft Sep 10 '20 at 19:40
  • Yes, I'm using the consumption plan. Not sure if SDK version is the same thing as runtime version? The runtime version is displayed on the function app's Overview blade in the portal, on the right-hand side. And when I checked back just now, it's 3.0.14413.0 so is clearly being updated quite often. – Stuart J Cuthbertson Sep 13 '20 at 21:16
  • I left this answer fairly vague as I haven't conclusively resolved my specific issue yet. If it happens again I'll probably recreate the function app with `--runtime-version` specified to see if that fixes it. But I can't justify spending that time if this were just a one-off hour outage. – Stuart J Cuthbertson Sep 13 '20 at 21:19
  • As your function scales, it is deployed to new servers. Your sdk version is determined by the server on which the function runs. It isn't something you control. Glad you are no longer experiencing the error. I would recommend using app insights query and correlate (or not) whether your error occured on an sdk prior to the fix or not. – Troy Witthoeft Sep 14 '20 at 00:46
  • 1
    Aha, there's some mixed terminology going on here. The SDK version when the errors occurred was indeed 3.0.14287.0 - App Insights confirms this. It seems the Azure Portal sometimes refers to this version track as `Runtime version`, but also uses the `--runtime-version` CLI switch to refer to _Python_ interpreter versions. – Stuart J Cuthbertson Sep 21 '20 at 17:55
  • The third point of `Authentication`: if you are using a wrong key for Authentication will also yield this error. Refresh you *master* key – Juan Javier Triff Cabanas Aug 22 '23 at 17:25
3

Same error but different technology, environment, and root cause. Technology Net 5, target system windows. In my case, I was using dependency injection to add a few services, I was getting one parameter from the environment variables inside the .ConfigureServices() section, but when I deployed I forget to add the variable to the application settings in azure, because of that I was getting this weird error.

Jorge Cortes P.
  • 31
  • 2
2

This error is most likely github issue #4384. This bug was identified, and a fix was released mid-june 2020. Apps running on version 3.0.14063 or greater should be fine. List of versions is here.

You can use azure application insights to check your version. KUSTO Query the logs. The exception table, azure SDK column has your version.

If you are on the dedicated App Service plan, you may be able to "pull" the latest version from Microsoft by deleting and redeploying your app. If you are on consumption plan, then you may need to wait for this bugfix to rollout to all servers.

Dharman
  • 30,962
  • 25
  • 85
  • 135
Troy Witthoeft
  • 2,498
  • 2
  • 28
  • 37
  • My interpretation of GitHub #4384 is that it only relates to use of the VS Code extension for Azure Functions. This would only affect local execution on a developer machine, surely? Whereas this question is about a problem that only happens on Azure, not on dev machines. Can you clarify? – Stuart J Cuthbertson Sep 08 '20 at 16:45
  • 1
    The referenced github thread is inside the https://github.com/Azure/azure-functions-host repo. That repo - those libraries - are the runtime used by Azure Functions AND your local CLI. While the topic of conversation in the #4384 thread revolves around reproducing it locally in VSCode, the local reproduction does not preclude the same issue from being present in in Azure. It was. Both OP and my organization observed it. We engaged MS and repo maintainers. They upped our Azure environments to SDK 3.0.14063+ and the problem was resolved. Thanks. – Troy Witthoeft Sep 09 '20 at 21:24
1

This is due to SDK version, I would suggest to deploy fresh function App in Azure and deploy your code there. 2 things to check :

  1. Make sure your local function app SDK version matches with Azure function app.
  2. Check python version both side.
Harish
  • 327
  • 4
  • 14
1

Took me a while to find the cause as well, but it was related to me installing a version of protobuf explicitly which conflicted with what was used by Azure Functions. Fair, there was a warning about that in the docs. How I found it: went to <your app name>.scm.azurewebsites.net/api/logstream and looked for any errors I could find.

Dmitry Smirnov
  • 462
  • 8
  • 22