1

BACKGROUND:

I'm trying to build a postgres-compatible rust server using actix-web & sqlx into a docker container. It's a basic RESTful API for a mock blog, just so I can establish a basic scaffolding for a useful server architecture that I can use as a basis for future projects.

It builds & runs perfectly well outside the docker container, and I've isolated the issue to two dependencies: actix-web & sqlx-- unfortunately the two core dependencies for the project. When I remove these two dependencies from Cargo.toml, the container builds within seconds.

For my Dockerfile I'm using a multi-stage build based upon this tutorial, using cargo chef to cache the build.

PROBLEM:

When running docker build -t server ., after ~30-45 minutes cargo will throw numerous warning: spurious network errors and timeout's, despite building normally outside of docker.

NOTES:

  • I am running MacOs on a laptop with a 2.3 GHz 8-Core Intel Core i9 processor, and 32 GB 2400 MHz DDR4 memory
  • My internet connection is stable, running at 442.33 Mbps download & 21.50 Mbps upload through a VPN (the issue persists with VPN disabled as well)
  • I am running Docker v20.10.21, via homebrew
  • The same errors are thrown even with a simple single-stage build
  • Issue mirrored here on GitHub

Command:

From server: docker build -t server .

Or from root: docker-compose up

Code:

server/Cargo.toml

[package]
name = "actix-sqlx-docker"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
actix-web = "4.2.1"
chrono = { version = "0.4.23", features = [ "serde" ] }
dotenv = "0.15.0"
serde = { version = "1.0.150", features = [ "derive" ] }
serde_json = "1.0.89"
sqlx = { version = "0.6.2", features = [ "runtime-actix-native-tls", "postgres", "chrono", "time", "uuid"] }
uuid = { version = "1.2.2", features = [ "v4", "fast-rng", "macro-diagnostics", "serde" ] }

server/Dockerfile

# generate a recipe file for dependencies
FROM --platform=linux/amd64 rust as planner
WORKDIR /usr/src/app
RUN cargo install cargo-chef
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# build dependencies
FROM --platform=linux/amd64 rust as cacher
WORKDIR /usr/src/app
RUN cargo install cargo-chef
COPY --from=planner /usr/src/app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

# use rust docker image as builder
FROM --platform=linux/amd64 rust as builder
COPY . /usr/src/app
WORKDIR /usr/src/app
COPY --from=cacher /usr/src/app/target target
COPY --from=cacher /usr/local/cargo /usr/local/cargo

FROM --platform=linux/amd64 gcr.io/distroless/cc-debian11 as development

COPY --from=builder /usr/src/app/target/release/actix-sqlx-docker /usr/src/test-app/actix-sqlx-docker
WORKDIR /usr/src/app

CMD ["./actix-sqlx-docker"]

Errors:

Detailed view of errors can be viewed here

Extraneous error messages left out to conserve character count. See in full in linked GitHub issue.

Partial error from multi-stage build:

[+] Building 1919.8s (16/20)
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 913B                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 77B                                                                                                                                                                       0.0s
 => [internal] load metadata for gcr.io/distroless/cc-debian11:latest                                                                                                                                  3.0s
 => [internal] load metadata for docker.io/library/rust:latest                                                                                                                                         3.4s
 => [auth] library/rust:pull token for registry-1.docker.io                                                                                                                                            0.0s
 => [development 1/3] FROM gcr.io/distroless/cc-debian11@sha256:101c26286ea36b68200ff94cf95ca9dbde3329c987738cba3ba702efa3465f6f                                                                       1.6s
 => => resolve gcr.io/distroless/cc-debian11@sha256:101c26286ea36b68200ff94cf95ca9dbde3329c987738cba3ba702efa3465f6f                                                                                   0.0s
 => => sha256:a1f1879bb7de17d50f521ac7c19f3ed9779ded79f26461afb92124ddb1ee7e27 817.19kB / 817.19kB                                                                                                     0.6s
 => => sha256:101c26286ea36b68200ff94cf95ca9dbde3329c987738cba3ba702efa3465f6f 1.67kB / 1.67kB                                                                                                         0.0s
 => => sha256:e8c6063fa88722e5fc86eb631b41c0a46d7c8503aa2433298c59b07d3b731fbf 753B / 753B                                                                                                             0.0s
 => => sha256:7d62b76a133b357bbc8eca89d5aa845e8b807e31980789db995c851ca100a61c 776B / 776B                                                                                                             0.0s
 => => sha256:fc251a6e798157dc3b46fd265da72f39cd848e3f9f4a0b28587d1713b878deb9 795.83kB / 795.83kB                                                                                                     0.5s
 => => sha256:fda4ba87f6fbeebb651625814b981d4100a98c224fef80d562fb33853500f40e 8.00MB / 8.00MB                                                                                                         1.1s
 => => extracting sha256:fc251a6e798157dc3b46fd265da72f39cd848e3f9f4a0b28587d1713b878deb9                                                                                                              0.2s
 => => extracting sha256:fda4ba87f6fbeebb651625814b981d4100a98c224fef80d562fb33853500f40e                                                                                                              0.3s
 => => extracting sha256:a1f1879bb7de17d50f521ac7c19f3ed9779ded79f26461afb92124ddb1ee7e27                                                                                                              0.1s
 => [planner 1/5] FROM docker.io/library/rust@sha256:0067330b7e0eacacc5c32f21b720607c0cd61eda905c8d55e6a745f579ddeee9                                                                                 21.9s
 => => resolve docker.io/library/rust@sha256:0067330b7e0eacacc5c32f21b720607c0cd61eda905c8d55e6a745f579ddeee9                                                                                          0.0s
 => => sha256:0067330b7e0eacacc5c32f21b720607c0cd61eda905c8d55e6a745f579ddeee9 988B / 988B                                                                                                             0.0s
 => => sha256:5e0d0367026b13e8f26d3993fcd0d880da32a67ed45b2f3dd2467ac97de3077c 6.42kB / 6.42kB                                                                                                         0.0s
 => => sha256:3406614c4f79d01038ac6d7c608f856ed3a77f545c9ab7f521dddc8905ea2e63 1.59kB / 1.59kB                                                                                                         0.0s
 => => sha256:32de3c850997ce03b6ff4ae8fb00b34b9d7d7f9a35bfcdb8538e22cc7b77c29d 55.03MB / 55.03MB                                                                                                       4.4s
 => => sha256:fa1d4c8d85a4e064e50cea74d4aa848dc5fc275aef223fcc1f21fbdb1b5dd182 5.16MB / 5.16MB                                                                                                         1.6s
 => => sha256:c796299bbbddc7aeada9539a4e7874a75fa2b6ff421f8d5ad40f227b40ab4d86 10.88MB / 10.88MB                                                                                                       1.9s
 => => sha256:81283a9569ad5e90773f038daedd0d565810ca5935eec8f53b8bcb6a199030d6 54.58MB / 54.58MB                                                                                                       7.6s
 => => sha256:60b38700e7fb2cdfac79b15e4c1691a80fe6b4101c7b7fea66b9e7cd64d961cf 196.88MB / 196.88MB                                                                                                     7.9s
 => => sha256:5e868601ee3b6add93e6e2e5794ac5d9e29de9eaeb7920466e30c4e92cd06e7f 175.92MB / 175.92MB                                                                                                    11.0s
 => => extracting sha256:32de3c850997ce03b6ff4ae8fb00b34b9d7d7f9a35bfcdb8538e22cc7b77c29d                                                                                                              2.7s
 => => extracting sha256:fa1d4c8d85a4e064e50cea74d4aa848dc5fc275aef223fcc1f21fbdb1b5dd182                                                                                                              0.4s
 => => extracting sha256:c796299bbbddc7aeada9539a4e7874a75fa2b6ff421f8d5ad40f227b40ab4d86                                                                                                              0.4s
 => => extracting sha256:81283a9569ad5e90773f038daedd0d565810ca5935eec8f53b8bcb6a199030d6                                                                                                              2.6s
 => => extracting sha256:60b38700e7fb2cdfac79b15e4c1691a80fe6b4101c7b7fea66b9e7cd64d961cf                                                                                                              5.6s
 => => extracting sha256:5e868601ee3b6add93e6e2e5794ac5d9e29de9eaeb7920466e30c4e92cd06e7f                                                                                                              4.9s
 => [internal] load build context                                                                                                                                                                      0.0s
 => => transferring context: 62.71kB                                                                                                                                                                   0.0s
 => [builder 2/5] COPY . /usr/src/app                                                                                                                                                                  3.2s
 => [planner 2/5] WORKDIR /usr/src/app                                                                                                                                                                 3.2s
 => [planner 3/5] RUN cargo install cargo-chef                                                                                                                                                        52.6s
 => [builder 3/5] WORKDIR /usr/src/app                                                                                                                                                                 0.0s
 => [planner 4/5] COPY . .                                                                                                                                                                             0.0s
 => [planner 5/5] RUN cargo chef prepare --recipe-path recipe.json                                                                                                                                     0.3s
 => [cacher 4/5] COPY --from=planner /usr/src/app/recipe.json recipe.json                                                                                                                              0.0s
 => ERROR [cacher 5/5] RUN cargo chef cook --release --recipe-path recipe.json                                                                                                                      1838.2s
------
 > [cacher 5/5] RUN cargo chef cook --release --recipe-path recipe.json:
... // EXTRANEOUS MESSAGES LEFT OUT TO CONSERVE CHARACTER COUNT
#16 1687.7 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `subtle v2.4.1` within 30s)
#16 1717.8 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `thiserror v1.0.38` failed to transfer more than 10 bytes in 30s)
#16 1717.8 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `thiserror-impl v1.0.38` within 30s)
#16 1747.9 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `time v0.1.45` failed to transfer more than 10 bytes in 30s)
#16 1747.9 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `time v0.3.17` within 30s)
#16 1778.0 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `unicode-bidi v0.3.8` failed to transfer more than 10 bytes in 30s)
#16 1778.0 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `unicode_categories v0.1.1` within 30s)
#16 1808.1 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `url v2.3.1` failed to transfer more than 10 bytes in 30s)
#16 1808.1 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `uuid-macro-internal v1.2.2` within 30s)
#16 1838.2 error: failed to download from `https://crates.io/api/v1/crates/derive_more/0.99.17/download`
#16 1838.2
#16 1838.2 Caused by:
#16 1838.2   [28] Timeout was reached (download of `derive_more v0.99.17` failed to transfer more than 10 bytes in 30s)
#16 1838.2 thread 'main' panicked at 'Exited with status code: 101', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/cargo-chef-0.1.50/src/recipe.rs:176:27
#16 1838.2 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
------
executor failed running [/bin/sh -c cargo chef cook --release --recipe-path recipe.json]: exit code: 101

Partial error from single-staged build:

[+] Building 2668.9s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 1.10kB                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 77B                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/rust:latest                                                                                                                                         1.2s
 => [internal] load build context                                                                                                                                                                      0.0s
 => => transferring context: 62.71kB                                                                                                                                                                   0.0s
 => [1/4] FROM docker.io/library/rust@sha256:0067330b7e0eacacc5c32f21b720607c0cd61eda905c8d55e6a745f579ddeee9                                                                                         22.8s
 => => resolve docker.io/library/rust@sha256:0067330b7e0eacacc5c32f21b720607c0cd61eda905c8d55e6a745f579ddeee9                                                                                          0.0s
 => => sha256:fa1d4c8d85a4e064e50cea74d4aa848dc5fc275aef223fcc1f21fbdb1b5dd182 5.16MB / 5.16MB                                                                                                         0.7s
 => => sha256:3406614c4f79d01038ac6d7c608f856ed3a77f545c9ab7f521dddc8905ea2e63 1.59kB / 1.59kB                                                                                                         0.0s
 => => sha256:32de3c850997ce03b6ff4ae8fb00b34b9d7d7f9a35bfcdb8538e22cc7b77c29d 55.03MB / 55.03MB                                                                                                       3.9s
 => => sha256:c796299bbbddc7aeada9539a4e7874a75fa2b6ff421f8d5ad40f227b40ab4d86 10.88MB / 10.88MB                                                                                                       2.4s
 => => sha256:0067330b7e0eacacc5c32f21b720607c0cd61eda905c8d55e6a745f579ddeee9 988B / 988B                                                                                                             0.0s
 => => sha256:5e0d0367026b13e8f26d3993fcd0d880da32a67ed45b2f3dd2467ac97de3077c 6.42kB / 6.42kB                                                                                                         0.0s
 => => sha256:81283a9569ad5e90773f038daedd0d565810ca5935eec8f53b8bcb6a199030d6 54.58MB / 54.58MB                                                                                                       3.5s
 => => sha256:60b38700e7fb2cdfac79b15e4c1691a80fe6b4101c7b7fea66b9e7cd64d961cf 196.88MB / 196.88MB                                                                                                    12.0s
 => => sha256:5e868601ee3b6add93e6e2e5794ac5d9e29de9eaeb7920466e30c4e92cd06e7f 175.92MB / 175.92MB                                                                                                     8.2s
 => => extracting sha256:32de3c850997ce03b6ff4ae8fb00b34b9d7d7f9a35bfcdb8538e22cc7b77c29d                                                                                                              2.5s
 => => extracting sha256:fa1d4c8d85a4e064e50cea74d4aa848dc5fc275aef223fcc1f21fbdb1b5dd182                                                                                                              0.3s
 => => extracting sha256:c796299bbbddc7aeada9539a4e7874a75fa2b6ff421f8d5ad40f227b40ab4d86                                                                                                              0.4s
 => => extracting sha256:81283a9569ad5e90773f038daedd0d565810ca5935eec8f53b8bcb6a199030d6                                                                                                              2.6s
 => => extracting sha256:60b38700e7fb2cdfac79b15e4c1691a80fe6b4101c7b7fea66b9e7cd64d961cf                                                                                                              5.7s
 => => extracting sha256:5e868601ee3b6add93e6e2e5794ac5d9e29de9eaeb7920466e30c4e92cd06e7f                                                                                                              4.8s
 => [2/4] WORKDIR /usr/src/app                                                                                                                                                                         2.9s
 => [3/4] COPY . .                                                                                                                                                                                     0.0s
 => ERROR [4/4] RUN cargo build --release                                                                                                                                                           2641.9s
------
 > [4/4] RUN cargo build --release:
... // EXTRANEOUS MESSAGES LEFT OUT TO CONSERVE CHARACTER COUNT 
#8 2430.7 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `unicode-bidi v0.3.8` within 30s)
#8 2460.8 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `unicode-ident v1.0.6` failed to transfer more than 10 bytes in 30s)
#8 2460.8 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `unicode-normalization v0.1.22` within 30s)
#8 2491.0 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `unicode-segmentation v1.10.0` failed to transfer more than 10 bytes in 30s)
#8 2491.0 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `unicode_categories v0.1.1` within 30s)
#8 2521.1 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `url v2.3.1` failed to transfer more than 10 bytes in 30s)
#8 2521.1 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `uuid v1.2.2` within 30s)
#8 2551.3 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `uuid-macro-internal v1.2.2` failed to transfer more than 10 bytes in 30s)
#8 2551.3 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `version_check v0.9.4` within 30s)
#8 2581.4 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `whoami v1.2.3` failed to transfer more than 10 bytes in 30s)
#8 2581.4 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `zstd v0.11.2+zstd.1.5.2` within 30s)
#8 2611.6 warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `zstd-safe v5.0.2+zstd.1.5.2` failed to transfer more than 10 bytes in 30s)
#8 2611.6 warning: spurious network error (1 tries remaining): [28] Timeout was reached (failed to download any data for `zstd-sys v2.0.4+zstd.1.5.2` within 30s)
#8 2641.8 error: failed to download from `https://crates.io/api/v1/crates/actix-http/3.2.2/download`
#8 2641.8
#8 2641.8 Caused by:
#8 2641.8   [28] Timeout was reached (download of `actix-http v3.2.2` failed to transfer more than 10 bytes in 30s)
------
executor failed running [/bin/sh -c cargo build --release]: exit code: 101
  • 1
    When including links in Stack Overflow questions you should "also copy the code into the question itself" as per [ask]. The same goes for the errors. But make sure it's a [mre]. – cafce25 Dec 26 '22 at 22:48
  • @cafce25 Thanks. Since the error messages were too large, I could only share snippets. – myssyng lynx Dec 27 '22 at 01:00

1 Answers1

0

Looks like I solved the issue in 040f1b8.

Sometimes during the cacher stage, it still stalls during the download step, but aborting & re-running the build works within a few tries. Could be an issue when running this via a cloud provider but it works for now.

Two things seemed to help:

  1. cargo-chef seems to have trouble with dependency-name: { version = "1", features = [ "foo", "bar" ] } syntax, so I removed that syntax from Cargo.toml.
[dependencies]
actix = "0.13"
actix-web = "4"
dotenv = "0.15.0"
serde_json = "1.0.89"

[dependencies.chrono]
version = "0.4.23"
features = [ "serde" ]

[dependencies.serde]
version = "1.0.150"
features = [ "derive" ]

[dependencies.sqlx]
version = "0.6.2"
features = [ "runtime-async-std-native-tls", "postgres", "chrono", "time", "uuid" ]

[dependencies.uuid]
version = "1.2.2"
features = [ "v4", "fast-rng", "macro-diagnostics", "serde" ]
  1. The entire rust package was overkill, so I ended up using an alpine distro instead. Also specifying the --target flag in the cacher stage seemed to help. I don't think my previous Dockerfile would have run anyway, since the user wasn't configured.
# generic lite rust container with musl & openssl
FROM --platform=linux/amd64 rust:alpine3.17 as rust-alpine
RUN apk add --no-cache musl-dev openssl-dev
WORKDIR /usr/src/app

# generic chef to minimize redundancy
FROM rust-alpine as chef
RUN cargo install cargo-chef

# generate a recipe file for dependencies
FROM chef as planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# build & cache dependencies
FROM chef as cacher
COPY --from=planner /usr/src/app/recipe.json recipe.json
RUN cargo chef cook --release --target x86_64-unknown-linux-musl --recipe-path recipe.json

# use rust:alpine3.17 docker image as builder
FROM rust-alpine as builder
# create user
ENV USER=dev
ENV UID=1337
RUN adduser \
  --disabled-password \
  --gecos "" \
  --home "/nonexistent" \
  --shell "/sbin/nologin" \
  --no-create-home \
  --uid "${UID}" \
  "${USER}"
# copy build image
COPY . .
COPY --from=cacher /usr/src/app/target target
COPY --from=cacher /usr/local/cargo /usr/local/cargo
RUN cargo build --release

# run executable from distroless build
FROM --platform=linux/amd64 gcr.io/distroless/cc-debian11 as development
# copy user from builder
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group
# copy executable from builder
COPY --from=builder /usr/src/app/target/release/actix-sqlx-docker /usr/local/bin/actix-sqlx-docker
# set user
USER dev:dev
# run executable
CMD ["/usr/local/bin/actix-sqlx-docker"]