Rust musl Docker image cannot find Cargo

Question

I'm trying to get Rust running in Docker to use it for 32-bit musl builds. Ever since I updated it to use the new URL to pull rustup, I'm hitting this issue when running the container interactively using bash:

root@2c3549fe3169:/sample# cargo
error: command failed: 'cargo'
info: caused by: No such file or directory (os error 2)

The weird thing is, I can see the executables

root@2c3549fe3169:/sample# ls -l /root/.cargo/bin/
total 101440
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 cargo
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 cargo-clippy
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 cargo-fmt
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 rls
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 rust-gdb
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 rust-lldb
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 rustc
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 rustdoc
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 rustfmt
-rwxr-xr-x 10 root root 10383380 Feb 17 21:34 rustup
root@2c3549fe3169:/sample# date
Sun Feb 17 21:34:21 UTC 2019
root@2c3549fe3169:/sample# file /root/.cargo/bin/cargo
/root/.cargo/bin/cargo: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.9, with debug_info, not stripped
root@2c3549fe3169:/sample# cargo
error: command failed: 'cargo'
info: caused by: No such file or directory (os error 2)

It is installed via:

RUN curl https://sh.rustup.rs -sSf | sh -s -- \
--default-toolchain 1.32.0 \
-y && \
~/.cargo/bin/rustup target add i686-unknown-linux-musl && \
echo "[build]\ntarget = \"i686-unknown-linux-musl\"" > ~/.cargo/config

I can see the file but I cannot seem to run it, even when I switch into that directory:

root@2c3549fe3169:~/.cargo/bin# ./cargo
error: command failed: 'cargo'
info: caused by: No such file or directory (os error 2)

This is what I see when running ldd:

root@4e21c8d37266:/volume# ldd /root/.cargo/bin/cargo
linux-gate.so.1 (0xf7f41000)
libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf774c000)
librt.so.1 => /lib/i386-linux-gnu/librt.so.1 (0xf7742000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf7723000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf7705000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7529000)
libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xf7427000)
/lib/ld-linux.so.2 (0xf7f43000)

This is my complete Dockerfile

FROM i386/ubuntu

RUN apt-get update && apt-get install -y \
  cmake \
  curl \
  file \
  git \
  g++ \
  python \
  make \
  nano \
  ca-certificates \
  xz-utils \
  musl-tools \
  pkg-config \
  apt-file \
  xutils-dev \
  --no-install-recommends && \
  rm -rf /var/lib/apt/lists/*


RUN curl https://sh.rustup.rs -sSf | sh -s -- \
  --default-toolchain 1.32.0 \
  -y && \
  ~/.cargo/bin/rustup target add i686-unknown-linux-musl && \
  echo "[build]\ntarget = \"i686-unknown-linux-musl\"" > ~/.cargo/config


# Compile C libraries with musl-gcc
ENV SSL_VER=1.0.2j \
    CURL_VER=7.52.1 \
    CC=musl-gcc \
    PREFIX=/usr/local \
    PATH=/usr/local/bin:$PATH \
    PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

RUN curl -sL http://www.openssl.org/source/openssl-$SSL_VER.tar.gz | tar xz && \
    cd openssl-$SSL_VER && \
    ./Configure no-shared --prefix=$PREFIX --openssldir=$PREFIX/ssl no-zlib -m32 linux-elf -fPIC -fno-stack-protector && \
    make depend 2> /dev/null && make -j$(nproc) && make install && \
    cd .. && rm -rf openssl-$SSL_VER

RUN curl https://curl.haxx.se/download/curl-$CURL_VER.tar.gz | tar xz && \
    cd curl-$CURL_VER && \
    ./configure --enable-shared=no --enable-static=ssl --enable-optimize --prefix=$PREFIX --host=i686-pc-linux-gnu CFLAGS=-m32 \
      --with-ca-path=/etc/ssl/certs/ --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt --without-ca-fallback && \
    make -j$(nproc) && make install && \
    cd .. && rm -rf curl-$CURL_VER

ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
    SSL_CERT_DIR=/etc/ssl/certs \
    OPENSSL_LIB_DIR=$PREFIX/lib \
    OPENSSL_INCLUDE_DIR=$PREFIX/include \
    OPENSSL_DIR=$PREFIX \
    OPENSSL_STATIC=1 \
    PATH=/usr/local/bin:/root/.cargo/bin:$PATH

RUN echo $PATH

And strace'ing the cargo binary as per comments:

root@156da6108ff8:~/.cargo/bin# strace -f -e trace=execve cargo               
execve("/root/.cargo/bin/cargo", ["cargo"], 0xfffdd8fc /* 20 vars */) = 0
execve("/root/.rustup/toolchains/1.32.0-x86_64-unknown-linux-gnu/bin/cargo", ["/root/.rustup/toolchains/1.32.0-"...], 0x57d95620 /* 25 vars */) = -1 ENOENT (No such file or directory)
error: command failed: 'cargo'
info: caused by: No such file or directory (os error 2)
+++ exited with 1 +++

Answer 1

So, here is the summary of our investigations.

The base image used for build is i386/ubuntu with 32-bit environment inside, however, this image does nothing to appropriately mask the results of uname(2) calls (by having something like setarch linux32 as entrypoint), so, when running on 64-bit system (your case), any process inside the build container calling uname(2) or uname(1) sees x86_64 instead of i686. This is the root of the problem.

When you install cargo, you download and run the installation script, which detects the platform it runs on and downloads the appropriate version of rustup-init. The platform detection in this script recognizes correctly that it runs in 32-bit environment but on 64-bit kernel, so the script downloads 32-bit version of rustup-init. However, rustup-init decides that it runs on x86_64 (probably it sees x86_64 returned by uname(2), but does not perform the check for "32-bit environment on 64-bit kernel" case, like the installer script does). You can see it during the installation without -y:

Current installation options:

    default host triple: x86_64-unknown-linux-gnu
    default toolchain: stable
    modify PATH variable: yes

So, rustup installs 64-bit toolchain, and you end up with situation when calling cargo results in running 64-bit binary in 32-bit environment, so you see the error.

I still feel some sort of inconsistent behavior here, because both the installation script and rustup-init are parts of the same project and I don't really see the reason why should they detect the platform differently in the same environment (why can't rustup-init just be as smart as the installation script is?).

As @Shepmaster noticed, this is a known issue (Rustup installs 64bit compiler on a 32bit Docker instance). There are two workarounds possible:

• force the platform for the default toolchain by passing --default-host i686-unknown-linux-gnu to the installer;
• fool the installer by running it under setarch linux32 so that its call to uname(2) will see i686 instead of x86_64.

Personally, I would choose the first option, as it seems to be less hacky.