Diagnose `DllNotFoundException` for native library in a linux docker container

Question

I have a dotnet app that depends on the librdkafka-redist nuget. I want to run this app in a docker container based on an alpine image, built for the linux-arm64 platform.

At runtime I see:

Unhandled exception. System.DllNotFoundException: Failed to load the librdkafka native library.
   at Confluent.Kafka.Impl.Librdkafka.TrySetDelegates(List`1 nativeMethodCandidateTypes)
   at Confluent.Kafka.Impl.Librdkafka.LoadLinuxDelegates(String userSpecifiedPath)
   at Confluent.Kafka.Impl.Librdkafka.Initialize(String userSpecifiedPath)
   at Confluent.Kafka.Producer`2..ctor(ProducerBuilder`2 builder)
   at Confluent.Kafka.ProducerBuilder`2.Build()

The library appears to be packaged correctly; if I shell into a container I see this:

$ ls /app/runtimes/linux-arm64/native
librdkafka.so

but I can also tell that I'll have problems because alpine doesn't come packaged with glibc support:

$ ldd /app/runtimes/linux-arm64/native/librdkafka.so
    /lib/ld-musl-aarch64.so.1 (0xffffb4158000)
    libm.so.6 => /lib/ld-musl-aarch64.so.1 (0xffffb4158000)
    libdl.so.2 => /lib/ld-musl-aarch64.so.1 (0xffffb4158000)
    libpthread.so.0 => /lib/ld-musl-aarch64.so.1 (0xffffb4158000)
    libc.so.6 => /lib/ld-musl-aarch64.so.1 (0xffffb4158000)
Error loading shared library ld-linux-aarch64.so.1: No such file or directory (needed by /app/runtimes/linux-arm64/native/librdkafka.so)
Error relocating /app/runtimes/linux-arm64/native/librdkafka.so: __vsnprintf_chk: symbol not found
...

I can use the alpine gcompat package and patchelf to apparently resolve some of these (slightly anonymised Dockerfile):

FROM mcr.microsoft.com/dotnet/aspnet:6.0-alpine

RUN apk add patchelf
RUN apk add binutils
RUN apk add gcompat

RUN    adduser --disabled-password \
        --gecos "" \
        --no-create-home \
        --uid 10028 \
        myuser

USER myuser

COPY --chown=myuser:myuser . app/

RUN patchelf --remove-needed ld-linux-aarch64.so.1 /app/runtimes/linux-arm64/native/librdkafka.so && \
    patchelf --add-needed libgcompat.so.0 /app/runtimes/linux-arm64/native/librdkafka.so

ENV COREHOST_TRACE=1

ENTRYPOINT [ "dotnet", "app/MyApp.dll" ]

at which point I think I've resolved my native dependency problems:

$ ldd /app/runtimes/linux-arm64/native/librdkafka.so
    /lib/ld-musl-aarch64.so.1 (0xffffb5290000)
    libgcompat.so.0 => /lib/libgcompat.so.0 (0xffffb4d5b000)
    libm.so.6 => /lib/ld-musl-aarch64.so.1 (0xffffb5290000)
    libdl.so.2 => /lib/ld-musl-aarch64.so.1 (0xffffb5290000)
    libpthread.so.0 => /lib/ld-musl-aarch64.so.1 (0xffffb5290000)
    libc.so.6 => /lib/ld-musl-aarch64.so.1 (0xffffb5290000)
    libucontext.so.1 => /lib/libucontext.so.1 (0xffffb4d49000)
    libobstack.so.1 => /usr/lib/libobstack.so.1 (0xffffb4d36000)

but I still get DllNotFoundException at runtime. As I set COREHOST_TRACE=1 I can see that

Adding runtimeTargets native asset runtimes/linux-arm64/native/librdkafka.so rid=linux-arm64 assemblyVersion= fileVersion=0.0.0.0 from librdkafka.redist/1.9.2
...
Chose linux-arm64, so removing rid (win-x86) specific assets for package librdkafka.redist/1.9.2 and asset type native
Chose linux-arm64, so removing rid (win-x64) specific assets for package librdkafka.redist/1.9.2 and asset type native
Chose linux-arm64, so removing rid (osx-x64) specific assets for package librdkafka.redist/1.9.2 and asset type native
Chose linux-arm64, so removing rid (osx-arm64) specific assets for package librdkafka.redist/1.9.2 and asset type native
Chose linux-arm64, so removing rid (linux-x64) specific assets for package librdkafka.redist/1.9.2 and asset type native
...
Reconciling library librdkafka.redist/1.9.2
Parsed native deps entry 0 for asset name: librdkafka from package: librdkafka.redist, library version: 1.9.2, relpath: runtimes/linux-arm64/native/librdkafka.so, assemblyVersion , fileVersion 0.0.0.0
...
Processing native/culture for deps entry [librdkafka.redist, 1.9.2, runtimes/linux-arm64/native/librdkafka.so]
  Considering entry [librdkafka.redist/1.9.2/runtimes/linux-arm64/native/librdkafka.so], probe dir [], probe fx level:0, entry fx level:0
    Relative path query /app/runtimes/linux-arm64/native/librdkafka.so (skipped file existence check)
    Probed deps dir and matched '/app/runtimes/linux-arm64/native/librdkafka.so'
Adding to native path: /app/runtimes/linux-arm64/native/
...
Property NATIVE_DLL_SEARCH_DIRECTORIES = /app/runtimes/linux-arm64/native/:/usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.11/:

before it fails at startup.

So I think I've a) ascertained that the framework is aware (and hopefully attempting) to load the native library I think it is and b) ascertained that there are no missing dependencies that would mean the library can't be loaded.

Are there any more processes or steps I can go through to further diagnose what's going on? I think there are a series of dependencies that ldd isn't showing me -- on openssl, zlib etc which might be the problem, for example?

Answer 1

I did eventually solve this one. I used strace to examine exactly what the process was looking for, and found that it was looking for a library named alpine-librdkafka.so rather than the librdkafka.so indicated in the exception.

Some digging later and I found that Confluent.Kafka is actually switching internally on the distribution and looking for a file named for the distribution. Unfortunately, in the release we were using there was no alpine-librdkafka.so arm64 build.

We've added a build from source to our pipeline, and now all good. Incidentally, I believe the needed build was added around 1.9.0, but upgrading for us was more complicated than just building and packaging the build we needed from the source.