4

I edited the question, but my problem was manifesting itself in an alpine container. I now get the same problem in a container from scratch. This is the same question but a bit more narrowed down.

As the title describes, I have a working executable in an Ubuntu container that I use to build my app, but once I copy it in an Alpine container, I get Device or resource busy with the same executable and I'm a bit confused on what is going on.

Here's my dockerfile:

ARG UBUNTU_VERSION=20.04
FROM ubuntu:${UBUNTU_VERSION} as builder

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get -y install \
    curl \
    g++-10 \
    gcc-10 \
    ninja-build \
    unzip \
    zip \
    make \
    perl \
    pkg-config \
    git \
    gdb

RUN mkdir /opt/vcpkg \
    && curl -L -s "https://github.com/microsoft/vcpkg/tarball/8ce7b41302728ff6fc8bd377f572c4cbe8c64c1d" | tar --strip-components=1 -xz -C /opt/vcpkg \
    && /opt/vcpkg/bootstrap-vcpkg.sh

RUN mkdir /opt/cmake && \
    curl -s "https://cmake.org/files/v3.18/cmake-3.18.0-Linux-x86_64.tar.gz" | tar --strip-components=1 -xz -C /opt/cmake

RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10 && \
    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10

ENV PATH="/opt/cmake/bin:/opt/vcpkg:${PATH}"

RUN mkdir /build2
COPY src2 /source/src2
WORKDIR /build2
RUN cmake /source/src2 \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_GENERATOR=Ninja \
    -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_TOOLCHAIN_FILE=/opt/vcpkg/scripts/buildsystems/vcpkg.cmake 
RUN cmake --build . --target all

FROM scratch

COPY --from=builder /build2/http-beast-ssl /http-beast-ssl
COPY --from=builder /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
COPY --from=builder /usr/lib/x86_64-linux-gnu/libdl.so.2 /usr/lib/x86_64-linux-gnu/libdl.so.2
COPY --from=builder /usr/lib/x86_64-linux-gnu/libpthread.so.0 /usr/lib/x86_64-linux-gnu/libpthread.so.0
COPY --from=builder /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
COPY --from=builder /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
COPY --from=builder /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
COPY --from=builder /usr/lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc.so.6
COPY --from=builder /usr/lib/x86_64-linux-gnu/libm.so.6 /usr/lib/x86_64-linux-gnu/libm.so.6
COPY --from=builder /usr/share/ca-certificates /usr/share/ca-certificates
COPY --from=builder /etc/ssl/ /etc/ssl/

ENTRYPOINT ["/http-beast-ssl"]

When I run http-beast-ssl example.com 443 / in the ubuntu container, everything work fine. I get the HTML back in the console as expected.

When I do the same thing inside the container from scratch, then I get the error.

What is going on there and how can I fix it? Is there something needed to make networking work as expected in my container? What is missing there?

Here's the C++ source code as reference:

#include <boost/asio/ssl.hpp>
#include <boost/lexical_cast.hpp>
#include <openssl/cryptoerr.h>
#include <boost/beast/core.hpp>
#include <boost/beast/http.hpp>
#include <boost/beast/ssl.hpp>
#include <boost/beast/version.hpp>
#include <boost/asio/connect.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/asio/ssl/error.hpp>
#include <boost/asio/ssl/stream.hpp>
#include <boost/beast/version.hpp>
#include <openssl/opensslv.h>
#include <cstdlib>
#include <iostream>
#include <string>

namespace beast = boost::beast; // from <boost/beast.hpp>
namespace http = beast::http;   // from <boost/beast/http.hpp>
namespace net = boost::asio;    // from <boost/asio.hpp>
namespace ssl = net::ssl;       // from <boost/asio/ssl.hpp>
using tcp = net::ip::tcp;       // from <boost/asio/ip/tcp.hpp>

// Performs an HTTP GET and prints the response
int main(int argc, char** argv)
{
    std::cout << OPENSSL_VERSION_TEXT << std::endl;
    try
    {
        // Check command line arguments.
        if(argc != 4 && argc != 5)
        {
            std::cerr <<
                "Usage: http-client-sync-ssl <host> <port> <target> [<HTTP version: 1.0 or 1.1(default)>]\n" <<
                "Example:\n" <<
                "    http-client-sync-ssl www.example.com 443 /\n" <<
                "    http-client-sync-ssl www.example.com 443 / 1.0\n";
            return EXIT_FAILURE;
        }
        auto const host = argv[1];
        auto const port = argv[2];
        auto const target = argv[3];
        int version = argc == 5 && !std::strcmp("1.0", argv[4]) ? 10 : 11;

        // The io_context is required for all I/O
        net::io_context ioc;

        // The SSL context is required, and holds certificates
        ssl::context ctx(ssl::context::tlsv12_client);

        // This holds the root certificate used for verification
        ctx.set_default_verify_paths();

        // Verify the remote server's certificate
        ctx.set_verify_mode(ssl::verify_peer);

            // These objects perform our I/O
        tcp::resolver resolver(ioc);
        beast::ssl_stream<beast::tcp_stream> stream(ioc, ctx);

        // Set SNI Hostname (many hosts need this to handshake successfully)
        if(! SSL_set_tlsext_host_name(stream.native_handle(), host))
        {
            beast::error_code ec{static_cast<int>(::ERR_get_error()), net::error::get_ssl_category()};
            throw beast::system_error{ec};
        }

        // Look up the domain name
        auto const results = resolver.resolve(host, port);

        // Make the connection on the IP address we get from a lookup
        beast::get_lowest_layer(stream).connect(results);

        // Perform the SSL handshake
        stream.handshake(ssl::stream_base::client);

        // Set up an HTTP GET request message
        http::request<http::string_body> req{http::verb::get, target, version};
        req.set(http::field::host, host);
        req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);

        // Send the HTTP request to the remote host
        http::write(stream, req);

        // This buffer is used for reading and must be persisted
        beast::flat_buffer buffer;

        // Declare a container to hold the response
        http::response<http::dynamic_body> res;

        // Receive the HTTP response
        http::read(stream, buffer, res);

        // Write the message to standard out
        std::cout << res << std::endl;

        // Gracefully close the stream
        beast::error_code ec;
        stream.shutdown(ec);
        if(ec == net::error::eof)
        {
            // Rationale:
            // http://stackoverflow.com/questions/25587403/boost-asio-ssl-async-shutdown-always-finishes-with-an-error
            ec = {};
        }
        if(ec)
            throw beast::system_error{ec};

        // If we get here then the connection is closed gracefully
    }
    catch(boost::system::system_error const& e)
    {
        auto const& error = e.code();
        std::string err = error.message();
        if (error.category() == boost::asio::error::get_ssl_category()) {
            err = std::string(" (")
                +boost::lexical_cast<std::string>(ERR_GET_LIB(error.value()))+","
                +boost::lexical_cast<std::string>(ERR_GET_FUNC(error.value()))+","
                +boost::lexical_cast<std::string>(ERR_GET_REASON(error.value()))+") ";
    
            //ERR_PACK /* crypto/err/err.h */
            char buf[128];
            ::ERR_error_string_n(error.value(), buf, sizeof(buf));
            err += buf;
            std::cerr << err << std::endl;
        } else {
            std::cerr << err << std::endl;
        }
        return EXIT_FAILURE;
    }
    return EXIT_SUCCESS;
    
}

And here's my vcpkg.json and CMakeLists.txt files:

{
    "name": "mypackage",
    "version-string": "0.1.0-dev",
    "dependencies": [
        "boost-asio",
        "boost-beast",
        "openssl"
    ]
}
cmake_minimum_required(VERSION 3.18)
project(beast-test)

set(CMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY ON)
set(CMAKE_FIND_USE_CMAKE_SYSTEM_PATH OFF)
set(CMAKE_FIND_PACKAGE_PREFER_CONFIG ON)
set(CMAKE_FIND_USE_CMAKE_ENVIRONMENT_PATH OFF)

find_package(Boost 1.74 REQUIRED COMPONENTS system)
find_package(Threads REQUIRED)
find_package(OpenSSL REQUIRED)

add_executable(http-beast-ssl main2.cpp)
target_link_libraries(http-beast-ssl PRIVATE Threads::Threads Boost::system OpenSSL::SSL OpenSSL::Crypto)

At first I got the very same problem in an alpine container and that problem happened even if compiled using static linking.

When running curl https://example.com inside the alpine container it work as expected, but my app is still failing to make http requests.

Guillaume Racicot
  • 39,621
  • 9
  • 77
  • 141
  • If you can not use debugger, the only way is to comment the program line by line from the end and see what particular line of code spits the error... At least, I managed to do so using asio ssl. Sorry if my comment seems stupid. – Stepan Pavlov Nov 18 '20 at 10:23
  • Another suggestion - replace context ssl::verify_peer with ssl::verify_none... – Stepan Pavlov Nov 18 '20 at 10:46
  • @StepanPavlov the error happens at `resolver.resolve(host, port);` as far as I remember. – Guillaume Racicot Nov 18 '20 at 13:11
  • As I understand, the program is unable to open a socket, which is supplied by an operating system. And, you actually try to run the executable compiled in another OS. Can you compile it under Alpine and then run? – Stepan Pavlov Nov 18 '20 at 14:23
  • @StepanPavlov I can try, but would that help for a from scratch container? – Guillaume Racicot Nov 18 '20 at 15:09
  • I've just compiled and ran your code on my FreeBSD machine. It's read the example.com thoroughly... I used only your c++ code and CMakeLists.txt. Errors are absent. – Stepan Pavlov Nov 18 '20 at 15:41
  • @StepanPavlov same thing when I run it on my Ubuntu container. It works without error. It only don't work in the from scratch container, I'm sure it's missing something but I don't know what. – Guillaume Racicot Nov 18 '20 at 15:56
  • Actually I don't catch what exactly you mean under scratch container. Would you like to explain? – Stepan Pavlov Nov 18 '20 at 16:01
  • I create the container using the `FROM scratch` container, which is empty. In the dockerfile I posted, this is what I'm doing – Guillaume Racicot Nov 18 '20 at 16:04
  • Well, then we can assert that a mistake is in your docker file. – Stepan Pavlov Nov 18 '20 at 16:14
  • Did you try to use your entrypoint together with parameters `["/http-beast-ssl", "example.com 443 /"]`? [link](https://phoenixnap.com/kb/docker-cmd-vs-entrypoint#:~:text=CMD%20is%20an%20instruction%20that,container%20with%20a%20specific%20executable.) says that you cannot override them later in command line... – Stepan Pavlov Nov 18 '20 at 16:58
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/224743/discussion-between-guillaume-racicot-and-stepan-pavlov). – Guillaume Racicot Nov 18 '20 at 17:33

1 Answers1

2

I know the error message says something absolutely irrelevant but the problem here (as usual) is missing libraries. Add these lines to include DNS libs:

COPY --from=builder /usr/lib/x86_64-linux-gnu/libnss_dns.so.2 /usr/lib/x86_64-linux-gnu/libnss_dns.so.2
COPY --from=builder /usr/lib/x86_64-linux-gnu/libresolv.so.2 /usr/lib/x86_64-linux-gnu/libresolv.so.2

This is somewhat like a common pitfall with scratch. I recommend you to add all libraries while you are developing and testing image, then rewrite Dockerfile to leave only necessary ones. This way it is easier to find out that you forgot something.

UPDATE:

I debugged this by copying all libraries from builder step (Ubuntu) to my local machine (./libs) and mounting them into final (scratch) step:

version: "3.7"

services:
  test:
    build:
      context: .
    volumes:
    - ./libs:/usr/lib/x86_64-linux-gnu

After that it immediately started working. So I began removing copied libraries one by one until I made the image broken again. And that's how I found the required libraries. I'm sure there must be a better way to do this, I'm just not familiar with C++.

Full Dockerfile

ARG UBUNTU_VERSION=20.04
FROM ubuntu:${UBUNTU_VERSION} as builder

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get -y install \
    curl \
    g++-10 \
    gcc-10 \
    ninja-build \
    unzip \
    zip \
    make \
    perl \
    pkg-config \
    git \
    gdb

RUN mkdir /opt/vcpkg \
    && curl -L -s "https://github.com/microsoft/vcpkg/tarball/8ce7b41302728ff6fc8bd377f572c4cbe8c64c1d" | tar --strip-components=1 -xz -C /opt/vcpkg \
    && /opt/vcpkg/bootstrap-vcpkg.sh

RUN mkdir /opt/cmake && \
    curl -s "https://cmake.org/files/v3.18/cmake-3.18.0-Linux-x86_64.tar.gz" | tar --strip-components=1 -xz -C /opt/cmake

RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10 && \
    update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10

ENV PATH="/opt/cmake/bin:/opt/vcpkg:${PATH}"

RUN mkdir /build2
COPY src /source/src
WORKDIR /build2
RUN cmake /source/src \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_GENERATOR=Ninja \
    -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_TOOLCHAIN_FILE=/opt/vcpkg/scripts/buildsystems/vcpkg.cmake
RUN cmake --build . --target all

FROM scratch

COPY --from=builder /build2/http-beast-ssl /http-beast-ssl
COPY --from=builder /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
COPY --from=builder /usr/lib/x86_64-linux-gnu/libdl.so.2 /usr/lib/x86_64-linux-gnu/libdl.so.2
COPY --from=builder /usr/lib/x86_64-linux-gnu/libpthread.so.0 /usr/lib/x86_64-linux-gnu/libpthread.so.0
COPY --from=builder /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
COPY --from=builder /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
COPY --from=builder /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
COPY --from=builder /usr/lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc.so.6
COPY --from=builder /usr/lib/x86_64-linux-gnu/libm.so.6 /usr/lib/x86_64-linux-gnu/libm.so.6
COPY --from=builder /usr/lib/x86_64-linux-gnu/libnss_dns.so.2 /usr/lib/x86_64-linux-gnu/libnss_dns.so.2
COPY --from=builder /usr/lib/x86_64-linux-gnu/libresolv.so.2 /usr/lib/x86_64-linux-gnu/libresolv.so.2
COPY --from=builder /usr/share/ca-certificates /usr/share/ca-certificates
COPY --from=builder /etc/ssl/ /etc/ssl/

ENTRYPOINT ["/http-beast-ssl"]
anemyte
  • 17,618
  • 1
  • 24
  • 45
  • Great! I have some place to start now. Is there other libraries I should be aware of? I'm still getting the error. Is there a way I can debug stuff like that to know which libraries it tries to load? – Guillaume Racicot Nov 19 '20 at 14:36
  • Strange, that should be all you have to make it working. Here, I uploaded the image to Docker Hub, let's see if you have an error with it: `docker run --rm anemyte/boost-beast-ssl:latest example.com 443 /` – anemyte Nov 19 '20 at 15:12
  • I also added the full Dockerfile as well. @GuillaumeRacicot – anemyte Nov 19 '20 at 15:14
  • 1
    Oh I did build the container again and it seem to work. Wonderful!! Thank you very much. – Guillaume Racicot Nov 19 '20 at 16:17