1

When changing the EC2 instance type I ran into a problem. The machine had 3 Docker containers that needed to be restarted and after the restart their ports became unreachable.

What could be the issue and how should I go about obtaining other necessary debug information?

  • No changes were made to the Security Groups configuration within AWS, all the required ports are still enabled.

  • I am still able to SSH into the EC2 instance, but the ports used by Docker (80, 8181) are not accessible (connection timeout).

  • Within a web browser it doesn't matter if I'm trying to access a port that is being used or not, the browser behavior is always the same (loading indicator stops at the beginning, followed by timeout, nothing gets logged in e.g. Apache's access.log or error.log).

  • Within a web browser neither addressing the instance by its Public DNS (IPv4), IPv4 Public IP, or its original domain name works.

  • Restarting the instance or changing its type again doesn't help

I am able to ping/telnet/wget the ports used by the Docker containers from within the instance:

$ docker exec f227cf8d9481 wget 127.0.0.1:8181
converted 'http://127.0.0.1:8181' (ANSI_X3.4-1968) -> 'http://127.0.0.1:8181' (UTF-8)
--2018-01-15 23:49:10--  http://127.0.0.1:8181/
Connecting to 127.0.0.1:8181... connected.
HTTP request sent, awaiting response... 401 Unauthorized

But not from the outside (the IP address still gets resolved):

$ wget <aws-ip>.<aws-zone>.<instance>.amazonaws.com:8181
--2018-01-16 00:53:32--  http://<aws-ip>.<aws-zone>.<instance>.amazonaws.com:8181/
Resolving <aws-ip>.<aws-zone>.<instance>.amazonaws.com... xxx.xxx.xxx.xxx
Connecting to <aws-ip>.<aws-zone>.<instance>.amazonaws.com|xxx.xxx.xxx.xxx|:8181... failed: Operation timed out.
Retrying.

The docker containers are running and mapping between Docker ports is done correctly:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                    NAMES
f227cf8d9481        cloud9              "forever /cloud9/s..."   3 seconds ago       Up 2 seconds        0.0.0.0:8080-8081->8080-8081/tcp, 80/tcp, 0.0.0.0:8181->8181/tcp, 0.0.0.0:81->3000/tcp   my-cloud9
fa0d2bbce863        wordpress           "docker-entrypoint..."   59 minutes ago      Up 59 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                 goofy_torvalds
6ada961a5ea0        mysql               "docker-entrypoint..."   About an hour ago   Up About an hour    0.0.0.0:3306->3306/tcp

Iptables setting seems to have the Docker ports enabled:

$ sudo iptables --table nat --list
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  <aws-ip>.<aws-zone>.<instance>.internal/16  anywhere            
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:mysql
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:https
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:http
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:8181
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:tproxy
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:webcache
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:hbci

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp dpt:mysql to:<docker-ip>:3306
DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:<docker-ip>:443
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:<docker-ip>:80
DNAT       tcp  --  anywhere             anywhere             tcp dpt:8181 to:<docker-ip>:8181
DNAT       tcp  --  anywhere             anywhere             tcp dpt:tproxy to:<docker-ip>:8081
DNAT       tcp  --  anywhere             anywhere             tcp dpt:webcache to:<docker-ip>:8080
DNAT       tcp  --  anywhere             anywhere             tcp dpt:81 to:<docker-ip>:3000

No there doesn't seem to be any noticeable network activity (around 2KB every 5 mins) using the Monitoring tool of the EC2 instance. Except for the some spikes from the times I used SSH to log in:

enter image description here

Peter Gerhat
  • 111
  • 1
  • 4
  • Do pcaps show packets making it to the instance at all? – womble Jan 16 '18 at 02:10
  • No there doesn't seem to be any noticeable network activity (around 2KB every 5 mins) using the Monitoring tool of the EC2 instance. Except for the some spikes from the times I used SSH to log in. – Peter Gerhat Jan 16 '18 at 08:06
  • 1
    Are you able to reach docker containers from the docker host? Try something like ```curl -i http://127.0.0.1/``` and ```curl -i http://127.0.0.1:8181/``` Did you look at containers logs - ```docker logs fa0d2bbce863/f227cf8d9481/6ada961a5ea0``` ? – ALex_hha Jan 16 '18 at 10:37
  • Don't trust the graphs, check the pcaps. If the packets aren't making it to your instance, then the problem is in the security groups or network ACLs. – womble Jan 16 '18 at 10:47
  • 1
    Could it be that the expected ports were in use so other ones have been set? and these ports are not configured in the SG's? – Itai Ganot Jan 16 '18 at 13:00
  • @ALex_hha `curl -i http://127.0.0.1:8181/` gets picked up by the container. Log recovered using `docker logs` shows only internal requests by the docker host reaching the containers. – Peter Gerhat Jan 16 '18 at 22:00
  • @ItaiGanot You might have pointed me in the right direction, `lsof -i :8181` returns empty (port not assigned) on the host machine. But `docker ps` still shows the port as assigned (`0.0.0.0:8181->8181/tcp`). – Peter Gerhat Jan 16 '18 at 22:10
  • If possible, try to kill all the related containers, restart docker service, re-up the problematic container and retry. Also, do you see anything in the docker logs of the container? – Itai Ganot Jan 17 '18 at 09:10

1 Answers1

0

The issue was with the DNS and SSL certificate configuration, because the instance was configured to use HTTPS only..

After the change of the instance type, the new instance was automatically assigned a new URL, which had to be updated with both the DNS provider and the CA.

Peter Gerhat
  • 111
  • 1
  • 4