9

I have a docker compose file that starts up few containers including prometheus, alertmanager and grafana. These containers are not able to connect to internet. I have tried multiple solutions but to no avail. I am on a digitalocean ubuntu droplet.

My docker-compose file:

version: '3'

services:

    prometheus:
      image: prom/prometheus:v2.20.1
      container_name: prometheus
      ports:
        - 9090:9090
      volumes:
        - /data/prometheus:/prometheus
        - ./prometheus/:/etc/prometheus/
      restart: always
    
    alertmanager:
      image: prom/alertmanager:v0.21.0
      container_name: alertmanager
      ports:
        - 9093:9093
        - 6783:6783
      command:
        - '--log.level=debug'
        - '--config.file=/etc/alertmanager/alertmanager_config.yml'
        - '--storage.path=/alertmanager'
      volumes:
        - ./alertmanager:/etc/alertmanager
        - /data/alertmanager:/alertmanager
      restart: always


    grafana:
      image: grafana/grafana:7.1.5
      container_name: grafana
      ports:
        - 3000:3000
      volumes:
        - ./grafana.ini:/etc/grafana/grafana.ini
      restart: always

I have tried multiple things

  • Installed resolvconf and restarted docker service docker restart
  • Changed /etc/resolv.conf on host machine to point to google or openDNS servers.
  • Added DNS in /etc/docker/daemon.json and restarted docker
{
    "dns" : ["172.24.100.50", "8.8.8.8"]
}
  • Changed DNS nameserver inside the containers from
nameserver 127.0.0.11
options ndots:0

to

nameserver 127.0.0.11
nameserver 172.24.100.50
nameserver 8.8.8.8

Commands run inside the container

/alertmanager $ wget http://curl.haxx.se/download/curl-7.36.0.tar.gz
wget: bad address 'curl.haxx.se'
/alertmanager $ nslookup google.com
;; connection timed out; no servers could be reached

/alertmanager $

While sending alerts, alertmanager gives error:

lookup api.<my website>.com on 172.24.100.50:53: read udp 172.18.0.5:44178->172.24.100.50:53: i/o timeout"

I tried to run alertmanager on host network and it still doesn't work

docker run --net host -d prom/alertmanager:v0.21.0
docker exec -it <container_id> sh

/alertmanager $ cat /etc/resolv.conf 
nameserver 172.24.100.50
nameserver 8.8.8.8
/alertmanager $ ls
/alertmanager $ wget http://curl.haxx.se/download/curl-7.36.0.tar.gz
wget: bad address 'curl.haxx.se'
/alertmanager $ set vc
/alertmanager $ nslookup google.com
;; connection timed out; no servers could be reached

I have tried many options but haven't found the solution yet. Anyone who can help me with this? let me know if more details are required.

thecodeboxed
  • 420
  • 1
  • 6
  • 17
  • Try to ping google with the IP address to check if is a DNS problem or connection problem – Max Sep 26 '20 at 20:24
  • nslookup 172.217.163.46 outputs ;; connection timed out; no servers could be reached. I think it's a connection issue – thecodeboxed Sep 26 '20 at 21:10
  • Try this: `docker run -it ubuntu /bin/bash` then install ping with `apt update && apt install iputils-ping` then try to ping google by ip and by fqdn. If the problem persist you have to search the on your LAN. – Max Sep 27 '20 at 14:41

3 Answers3

1

I was able to solve the issue. It turns out in the digitalocean firewall for the droplet, the outbound traffic were blocked for UDP. Only TCP traffic was allowed. And hence the dns resolution was not working.

DNS uses TCP for Zone transfer and UDP for name queries either regular (primary) or reverse. UDP can be used to exchange small information whereas TCP must be used to exchange information larger than 512 bytes. DNS requires port 53 for name resolution and from the docker logs it can be seen port 53 is being used but since udp outbound traffic were blocked, dns was not working.

However, I did try to force docker to use TCP by setting dns_opt=use-vc setting. This didn't work. UDP traffic was allowed and now it is working.

thecodeboxed
  • 420
  • 1
  • 6
  • 17
  • 2
    Hello, I didn't understand how you solved the issue exactly. Is setting dns_opt=use-vc worked or not? if yes where did you set that? if not, how you did exactly to solve the problem. Thank you in advance – Sunshine Jun 17 '21 at 07:52
  • 1
    Your answer is not clear, Please describe in detail – Harun-Ur-Rashid Jul 13 '21 at 14:03
0

I ran into the same problem on Ubuntu LTS 22. In my case the problem was the docker installation was incorrect. I had installed docker on Ubuntu using the Snap app store and it didn't work.

I uninstalled docker and then installed it again, this time using the recommended method from the repository. The docker installation details for Ubuntu can be found here:

https://docs.docker.com/engine/install/ubuntu

FYI - I ran into the exact same problem on the Amazon Linux 2 machine due to a bad docker installation.

Hope it helps!

Razal Minhas
  • 1
  • 2
-1

It could be simply that your host iptables rule is blocking.
On your host machine, double check the output of iptables -L -v -n. Or temporarily even try something like iptables -P INPUT ACCEPT and then after confirming it work, revert it with iptables -P INPUT DROP. For the access to the outside world instead of local docker dns, replace INPUT with FORWARD.

smekkley
  • 62
  • 2