7

The following BASH script downloads and builds mpv (https://github.com/mpv-player/mpv-build) in a Docker container, and sends the file to the host using netcat (the host is listening using netcat):

HOSTIP="192.168.1.100"
PORT="62514"

nc -v -l $PORT > mpv &

sleep 1

sudo docker run ubuntu:14.04 /bin/bash -c "\
sed -i -e 's/# deb/deb/g' /etc/apt/sources.list; \
cat /etc/apt/sources.list; \
apt-get update; \
apt-get install --yes --force-yes git python-minimal; \
git clone https://github.com/mpv-player/mpv-build.git; \
cd mpv-build/; \
./update; \
apt-get install --yes --force-yes devscripts equivs; \
rm -f mpv-build-deps_*_*.deb; \
mk-build-deps -i -t \"apt-get --yes --force-yes --no-install-recommends\"; \
./build -j\$(nproc); \
cat mpv/build/mpv | nc $HOSTIP $PORT; \
echo Done"

#close any nc process that might be left running
nc $HOSTIP $PORT

So the script:

  1. Starts netcat (nc) on the host, listening for connections in the background
  2. Starts a Docker container, in which we clone the mpv repo, build mpv, use nc to connect to the listening socket on the host, and send the resulting binary back to the host

For some reason the resulting file is reduced in size, when received on the host side. Either it's 0 bytes or some multiple of 1024 bytes. The beginning seems intact, an x86_64 ELF executable.

Here's one packet capture where the docker container sends some of the mpv binary to the host, but where the listening nc (on the host) closes the connection prematurely (it sends a packet with the FIN flag set, a few milliseconds afters the connection is established):

docker-nc-packetdump

49152 bytes were transmitted here (it's always a multiple of 1024).

There's not always TCP retransmission errors. I made another capture with no errors, but still only a little was sent (24576) of the total 21818582 bytes the resulting mpv binary is.

Not sure what is happening here, why does nc on the listening/host side send a FIN TCP packet shortly after the connection is opened?

runeks
  • 131
  • 1
  • 7
  • Why are you using netcat for this? Just curious. Also, what OS/distributions are involved? – ewwhite Oct 17 '14 at 23:14
  • You may want to post on stackoverflow as well. They have more docker answers. I'm curious about what is going on here too. – seanmcl Oct 18 '14 at 13:19
  • @ewwhite I'm using netcat here because it seems like the simplest solution to my problem (transferring a file from a Docker instance to the host). Or at least it would be if it worked. – runeks Oct 19 '14 at 11:01
  • @runeks Do you have access to the Docker host? – ewwhite Oct 19 '14 at 11:03
  • @ewwhite Both Docker and host OS is Ubuntu 14.04. What do you mean by "access to the Docker host", exactly? (I'm a beginner with Docker) – runeks Oct 19 '14 at 11:05
  • @seanmcl Yeah I'm thinking of migrating it to stackoverflow. I can see they have a lot more Docker-related questions compared to serverfault. – runeks Oct 19 '14 at 11:06
  • Thinking about this issue, it became clear to me why this happens. Or at least I have a theory. netcat doesn't *actually* send data to the recipient. What netcat does is fill up a buffer in the NIC, which the NIC should then proceed to send, over the wire, to the recipient. I think what happens is that netcat fills up this buffer, exits, after which the container exits, before the NIC buffer content has been sent over the wire. – runeks Feb 13 '15 at 08:59

3 Answers3

0

I don't have a solution to the specific oddities of the netcat problem, but Docker does provide a facility to accomplish this.

You can use docker cp <containerId>:/path/to/file /host/path/to/file rather than mess with netcat.

MikeSchinkel
  • 113
  • 7
ewwhite
  • 197,159
  • 92
  • 443
  • 809
  • 1
    If you can incorporate this solution into my existing script, I'm accepting this answer. The problem is that somehow I need to trigger the copy after `./build` finishes, but before the script exits. So I still need a way for the Docker container to communicate to the host "I'm done building, copy the file now". This is why the `netcat` solution seems so much better: a listning socket is opened on the host before the Docker container starts, and when the Docker container is finished doing what it does, it connects to this socket and sends the file. – runeks Oct 19 '14 at 11:12
  • 1
    Wait a minute. Give me a few minutes. I have a solution that I think works, but I need to test it. – runeks Oct 19 '14 at 11:24
  • @runeks Heh, okay... – ewwhite Oct 19 '14 at 11:25
0

Using the docker cp command, the script can be rewritten like so:

HOSTIP="192.168.1.100"
PORT="62514"

set -e

CONTAINERID=$(sudo docker run -d ubuntu:14.04 /bin/bash -c "\
sed -i -e 's/# deb/deb/g' /etc/apt/sources.list; \
cat /etc/apt/sources.list; \
apt-get update; \
apt-get install --yes --force-yes git python-minimal; \
git clone https://github.com/mpv-player/mpv-build.git; \
cd mpv-build/; \
./update; \
apt-get install --yes --force-yes devscripts equivs; \
rm -f mpv-build-deps_*_*.deb; \
mk-build-deps -i -t \"apt-get --yes --force-yes --no-install-recommends\"; \
./build -j\$(nproc); \
echo \"From container: done building!\" | nc $HOSTIP $PORT; \
nc -v -l $PORT")

CONTAINERIP=$(sudo docker inspect $CONTAINERID|grep IPAddress|sed 's/.*IPAddress": "//'|sed 's/",$//')

echo "Started mpv-build container with Docker container ID: $CONTAINERID and IP: $CONTAINERIP" 
echo "Waiting to hear from container that the build has finished..."

#listen on host. wait for container to connect and disconnect
nc -l $PORT

#copy file from running container, while it's listening for a connection with nc
sudo docker cp $CONTAINERID:/mpv-build/mpv/build/mpv ./

#connect to the listening socket in the container to make the container finish running
echo "bye" | nc  $CONTAINERIP $PORT

echo "Done"

netcat is now only used as a sort of messaging mechanism, for the container to signal to the host when it's done building (so the host can initiate the copy), and for the host to signal to the container that it's done copying the file (so the container can exit).

runeks
  • 131
  • 1
  • 7
0

Another solution is to just use stream redirection, i.e.:

docker run /bin/sh -c '/my/build/command 1>&2 && cat /my/build/artifact' > artifact

For multiple files, add tar:

docker run /bin/sh -c '/my/build/command 1>&2 && tar -cf- -C /my/build/artifacts .' | tar -xf- -C artifacts
protomouse
  • 113
  • 4