ACME certificates timeout with traefik

Question

Hello!

I am having issues finding out the reason why I am getting this error. Tried googling it. It seems to be an issue with dns lookup from the container.

Error in traefik log:

time="2020-01-30T12:12:12+01:00" level=error msg="Unable to obtain ACME certificate for domains \"traefik.xyz.se\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:54773->127.0.0.11:53: i/o timeout" providerName=cloudflare.acme routerName=traefik-secure@docker rule="Host(`traefik.xyz.se`)"
time="2020-01-30T12:12:32+01:00" level=error msg="Unable to obtain ACME certificate for domains \"hivemq.xyz.se\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:53671->127.0.0.11:53: i/o timeout" rule="Host(`hivemq.xyz.se`)" providerName=cloudflare.acme routerName=hivemq-secure@docker

Unable to lookup google from within traefik container. Don't know if this is working as intended?

/o/a/traefik> docker exec -it traefik /bin/sh
/ # nslookup google.se
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'google.se': Try again
/ #

Traefik docker-compose.yaml

version: '3'

services:
  traefik:
    image: traefik:v2.1
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    environment:
      - CF_API_EMAIL=redacted
      - CF_API_KEY=redacted
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/config.yml:/config.yml:ro
    labels:
  - "traefik.enable=true"
  - "traefik.http.routers.traefik.entrypoints=http"
  - "traefik.http.routers.traefik.rule=Host(`traefik.xyz.se`)"
  - "traefik.http.middlewares.traefik-auth.basicauth.users=redacted"
  - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
  - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
  - "traefik.http.routers.traefik-secure.entrypoints=https"
  - "traefik.http.routers.traefik-secure.rule=Host(`traefik.xyz.se`)"
  - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
  - "traefik.http.routers.traefik-secure.tls=true"
  - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
  - "traefik.http.routers.traefik-secure.tls.domains[0].main=xyz.se"
  - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.xyz.se"
  - "traefik.http.routers.traefik-secure.service=api@internal"    
networks:
  proxy:
    external: true

data/traefik.yml:

api:
  dashboard: true
  debug: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yml

certificatesResolvers:
  cloudflare:
    acme:
      email: redacted
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 0
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

Service example (hivemq) docker-compose.yml:

version: "3"

services:
  hivemq:
    image: hivemq/hivemq4
    container_name: hivemq
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    ports:
      - 1883:1883
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.hivemq.entrypoints=http"
      - "traefik.http.routers.hivemq.rule=Host(`hivemq.xyz.se`)"
      - "traefik.http.routers.hivemq.middlewares=https-redirect@file"
      - "traefik.http.routers.hivemq-secure.middlewares=secured@file"
      - "traefik.http.routers.hivemq-secure.entrypoints=https"
      - "traefik.http.routers.hivemq-secure.rule=Host(`hivemq.xyz.se`)"
      - "traefik.http.routers.hivemq-secure.tls=true"
      - "traefik.http.routers.hivemq-secure.service=hivemq"
      - "traefik.http.services.hivemq.loadbalancer.server.port=8080"
      - "traefik.docker.network=proxy"
    networks:
      - internal
      - proxy

networks:
  proxy:
    external: true
  internal:
    external: false

I have also tried reinstalling docker-ce, didn't help.

score 2 · Answer 1 · answered Apr 26 '20 at 10:03

I had a similar issue and it was due to a bug of Docker: all my containers had lost their connection to the internet but they were all already removed for maintenance puprose so I couldn't see it.

In the logs, cannot get ACME client get directory means that Traefik cannot connect to Let's Encrypt url.

I fixed it by:

Removing Traefik stack
Pruning networks so traefik-public was removed
Restarting Docker service

If it's not enough, you can try these:

Try to restart the Docker Engine, which will reset any iptables rules (assuming you are using Docker on Linux)
Try to restart your whole machine
Try to disable (temporary) the firewall of your machine to verify that it fixes the issue

As mentioned here: https://community.containo.us/t/cannot-create-renew-acme-certificate-cannot-get-acme-client-get-directory/2469/2

I gave a rapid look around concerning Docker bugs about loosing connection and seems to be a mess, for years: https://github.com/moby/moby/issues/15172

score 0 · Answer 2 · answered Feb 28 '20 at 09:03

0

Not a docker specialist but I had a similar issue and fixed it by activating ipv6 on docker daemon :

% grep ipv6 /etc/docker/daemon.json
    "ipv6": true`

You need to reload docker daemon then

% sudo systemctl reload docker

answered Feb 28 '20 at 09:03

Stephane S.

21
2