0

I am trying to run a Docker image from inside Google Cloud Shell (i.e. on an courtesy Google Compute Engine instance) as follows:

docker run -d -p 20000-30000:10000-20000 -it <image-id> bash -c bash

Previous to this step, netstat -tuapn has reported the following:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8998          0.0.0.0:*               LISTEN      249/python      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:13080           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:13081           0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:34490         0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:13082           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:13083           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:13084           0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:34490         127.0.0.1:48161         ESTABLISHED -               
tcp        0    252 172.17.0.2:22           173.194.92.34:49424     ESTABLISHED -               
tcp        0      0 127.0.0.1:48161         127.0.0.1:34490         ESTABLISHED 15784/python    
tcp6       0      0 :::22                   :::*                    LISTEN      -

So it looks to me as if all the ports between 20000 and 30000 are available, but the run is nevertheless terminated with the following error message:

Error response from daemon: Cannot start container : failed to create endpoint on network bridge: Timed out proxy starting the userland proxy

What's going on here? How can I obtain more diagnostic information and ultimately solve the problem (i.e. get my Docker image to run with the whole port range available).

Alexey Alexandrov
  • 2,951
  • 1
  • 24
  • 24
Drux
  • 11,992
  • 13
  • 66
  • 116

1 Answers1

1

Opening up ports in a range doesn't currently scale well in Docker. The above will result in 10,000 docker-proxy processes being spawned to support each port, including all the file descriptors needed to support all those processes, plus a long list of firewall rules being added. At some point, you'll hit a resource limit on either file descriptors or processes. See issue 11185 on github for more details.

The only workaround when running on a host you control is to not allocate the ports and manually update the firewall rules. Not sure that's even an option with GCE. Best solution will be to redesign your requirements to keep the port range small. The last option is to bypass the bridge network entirely and run on the host network where there are no more proxies and firewall rules with --net=host. The later removes any network isolation you have in the container, so tends to be recommended against.

BMitch
  • 231,797
  • 42
  • 475
  • 450
  • +1 Thx for the explanation. I should be able to redesign my requirements to keep the port range small(er). But this seems independent from the problem at hand where Docker seems to complain about a port in the range 10000-20000 being taken, while `netstat` doesn't think so. How can I be sure this won't happen with a smaller port range too? – Drux Jul 17 '16 at 19:40
  • I have now redesigned my requirements (and Dockerfile) to employ a range of 10 ports instead of 10000 (which should do for now), and the problem has "magically" disappeared: `docker run` no longer produces errors. – Drux Jul 17 '16 at 21:03
  • "Timed out proxy starting the userland proxy" doesn't look like a complaint of a port being taken, rather it's saying it took too long to start a `docker-proxy` process. Could be too long since there were 9,999 other processes trying to simultaneously launch, or could be a side effect of running out of file descriptors or some other resource. – BMitch Jul 17 '16 at 21:31