Built Azure Function for Linux doesn't include dependency

Question

We have an Azure Function running on a Linux host.
Our app is a netcoreapp3.1. It runs fine, except for one issue which I can't explain.
The csproj file has always been configured like this (only a snippet):

<PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
    <UserSecretsId>...</UserSecretsId>
    <RunAnalyzersDuringBuild>false</RunAnalyzersDuringBuild>
  </PropertyGroup>
<ItemGroup>
<PackageReference Include="Google.Cloud.Asset.V1" Version="2.6.0" />
</ItemGroup>

There are many other packages, but this one is the one with the issue. On Windows everything works fine, all good. On Linux (or WSL2) the app also builds fine, the Functions Host starts and all seems well, until we hit code that uses the Google.Cloud.Asset.V1 package. This package references Grpc.Core and the code then fails with

System.Private.CoreLib: Exception while executing function: inventory. Grpc.Core: Error loading native library. Not found in any of the possible locations: /mnt/c/development/app/App.Functions/bin/Debug/netcoreapp3.1/bin/libgrpc_csharp_ext.x64.so,/mnt/c/development/app/App.Functions/bin/Debug/netcoreapp3.1/bin/runtimes/linux/native/libgrpc_csharp_ext.x64.so,/mnt/c/development/app/App.Functions/bin/Debug/netcoreapp3.1/bin/../../runtimes/linux/native/libgrpc_csharp_ext.x64.so.

It doesn't seem to make munch sense to me because this used to work, but nothing in the csproj changed recently, apart from other dependencies that were added, but are unrelated to this.
Checking in bin/Debug/netcoreapp3.1/bin/runtimes there's no linux only Windows.

I do however see this directory here, which doesn't seem to be in the search path in the error message though. This is bin/Debug/netcoreapp3.1/runtimes.

Does anybody know how I can get this to work again?
I tried adding <RuntimeIdentifier> or <RuntimeIdentifiers> into the csproj, but that didn't change anything.

The package is broken, you need to contact the package maintainers to fix it. — Ian Kemp, Jan 18 '21 at 12:34
@IanKemp: I see no sign that it's broken, in that it works on Linux in other environments. We've had issues with Azure Functions before in terms of managing dependencies. I don't know what Azure Functions is doing differently to a regular environment. — Jon Skeet, Jan 18 '21 at 13:13
@JonSkeet I parsed the question as "it used to work on Azure Functions but no longer does", how did you parse it? Obviously I'm missing something here... — Ian Kemp, Jan 18 '21 at 14:11
@IanKemp: Yes, the same package (and indeed the same version) used to work on Azure Functions but no longer does. That suggests to me that it's Azure Functions which has changed and broken things - so that suggests it's *not* the package that's broken. If you have reason to believe that it's genuinely the package's fault, I'd love to hear more details, as I'd like to improve things. But I really suspect it's an Azure Functions deployment issue. — Jon Skeet, Jan 18 '21 at 14:19
(It's not clear to me from the question what the screenshots are screenshots *of* though - a build on Windows? A build on Linux? Something else?) — Jon Skeet, Jan 18 '21 at 14:20
@JonSkeet Ah, since that pertinent information (Azure Functions version/package version) is omitted, I guess it could be interpreted either way. Also, the penny only just dropped for me that you're one of the authors of that package! No offence intended, it's just that with the limited info provided, "broken package" seemed like the most likely scenario. — Ian Kemp, Jan 18 '21 at 14:24
I would say that without any more information, we really just don't know. I would suggest it's best not to *assert* that something is broken when there isn't really enough evidence to say either way. I'm working with the question author on a GitHub issue about this as well... hoping to get more information so we can track what's going on. But as I say, gRPC-based client libraries work fine in more "normal" Linux environments. — Jon Skeet, Jan 18 '21 at 14:37
"On Linux (or WSL2) the app also builds fine, the Functions Host starts and all seems well, until we hit code that uses the Google.Cloud.Asset.V1 package." So just to check, you're able to reproduce this without actually deploying it in Azure Functions - it fails locally? Can you reproduce it *without* any other packages? A [mcve] would really help. — Jon Skeet, Jan 18 '21 at 14:38
I've reproduced this myself in Docker now - it looks like the problem is that the Azure Functions SDK (or runtime?) creates a local copy of the DLLs it needs inside a nested "bin" directory, but *doesn't* copy the native libraries. Will continue investigating, but probably tomorrow... — Jon Skeet, Jan 18 '21 at 16:08
@jonskeet sorry, offline over night. I informed the Azure Functions team as well. It seems that it doesn't matter where the app is built, because as it turns out we build on GH Actions Windows, but only where it's run. — David O'Brien, Jan 18 '21 at 21:23
Right. I don't know much about the build/deploy split in Azure Functions, but it definitely looks like the `bin/output/bin/runtimes/win` directory is created by `func start` (whether running in Windows or Linux) but the `linux` one isn't. I'm trying to work out whether this changed in a recent version of the Azure Functions SDK. — Jon Skeet, Jan 19 '21 at 07:48
Hmm... old version of the Azure Functions SDK doesn't help either. — Jon Skeet, Jan 19 '21 at 07:58

score 4 · Accepted Answer · answered Jan 19 '21 at 08:13

It looks like this is a problem that was fixed in Grpc.Core 2.34.0 (by this commit, I believe). If you add an explicit dependency on Grpc.Core 2.34.0, like this:

<PackageReference Include="Grpc.Core" Version="2.34.0" />

... that seems to fix it. I still don't know why the runtime was copied into the "old" expected place for Windows but not for Linux - that feels like it's an Azure Functions SDK issue somehow. But with Grpc.Core 2.34.0, the native extension loader "knows" where to find it in the parent bin/runtimes directory instead.

Built Azure Function for Linux doesn't include dependency

1 Answers1