6

I've set up a Concourse server (on bare metal), and everything is working swimmingly, except that I can't start any jobs and at this point I'm at a complete loss. Any help or suggestions would be greatly appreciated!

Taking the tutorial's hello world as an example, I get the following error:

$ fly -t tutorial execute -c task_hello_world.yml
executing build 55 at https://{{full hostname}}/builds/55 
initializing
resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to ping registry: 2 error(s) occurred:

* ping https: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
* ping http: Get http://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

errored

Grepping through the logs for any mention of docker, I get the following (and no other errors in the syslog):

Jan 11 03:39:11 {{host}} concourse[13704]: {"timestamp":"1515641951.825016260","source":"worker","message":"worker.garden.extract-resources.extract.already-extracted","log_level":1,"data":{"resource-type":"docker-image","session":"2.1.6"}}

Jan 11 03:41:55 {{host}} concourse[13613]: {"timestamp":"1515642115.660775423","source":"atc","message":"atc.create-build.do.task.image.failed-to-get-latest-image-version","log_level":2,"data":{"container":"ce9a7c49-210d-4cf7-6786-d27b1046c76b","error":"resource script '/opt/resource/check []' failed: exit status 1\n\nstderr:\nfailed to ping registry: 2 error(s) occurred:\n\n* ping https: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n* ping http: Get http://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n","session":"240.3.1.2"}}

I've started my web target with:

/usr/share/concourse/bin/concourse \
  web \
  --basic-auth-username {{username}} \
  --basic-auth-password {{password}} \
  --session-signing-key /usr/share/concourse/keys/session_signing_key \
  --tsa-host-key /usr/share/concourse/keys/tsa_host_key \
  --tsa-authorized-keys /usr/share/concourse/keys/authorized_keys \
  --bind-port 8080 \
  --tls-bind-port 8443 \
  --tls-key /etc/letsencrypt/live/{{full hostname}}/privkey.pem \
  --tls-cert /etc/letsencrypt/live/{{full hostname}}/fullchain.pem \
  --postgres-data-source postgres://concourse:concourse@localhost/atc \
  --external-url https://{{full hostname}}

...and my worker with:

/usr/share/concourse/bin/concourse \
  worker \
  --garden-dns-server 8.8.8.8 \
  --work-dir /usr/share/concourse/workspace \
  --tsa-host {{full hostname}} \
  --tsa-public-key /usr/share/concourse/keys/tsa_host_key.pub \
  --tsa-worker-private-key /usr/share/concourse/keys/worker_key

Docker appears to work fine (e.g., docker run hello-world), as does logging in with docker login. The worker nodes seem happy, and even register that they know about some containers (!?):

$ fly -t tutorial workers
name    containers platform  tags  team  state    version 
worker  3           linux    none  none  running  1.2

I have iptables routing ports 80 and 443 to 8080 and 8443, but there doesn't appear to be any difference if I turn this off and use the secondary ports. Curl seems to work fine, which I take to suggest that it's not a DNS issue:

$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

Again, any suggestions are appreciated!

Gabor Angeli
  • 5,729
  • 1
  • 18
  • 29
  • 1
    seems similar with this issue https://github.com/concourse/concourse/issues/374 – Fendi jatmiko Jan 11 '18 at 08:17
  • Yeah, I saw that, but it doesn't seem to help. I'm not running in a container or on a corp network, and DNS seems to work fine. For good measure, I edited resolv.conf to point to 8.8.8.8 (the error persists), and checked DNS again: `$ nslookup registry-1.docker.io Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: registry-1.docker.io Address: 34.200.28.105 ...` – Gabor Angeli Jan 11 '18 at 08:33
  • Situation is complicated. Do not add 8.8.8.8 to resolv.conf, add this flag to the concourse worker cli invocation: "--garden-dns-server 8.8.8.8". Does it work ? The full explanation is by reading the tens of tickets opened on concourse github for this problem... The UNAUTHORIZED you get when using curl is misleading, I went through the same rabbit hole :-) – marco.m Feb 28 '18 at 19:50
  • Did you manage to get anywhere with this in the end, @GaborAngeli? – Joe Oct 09 '18 at 09:50
  • No, I switched to self hosted Gitlab and life became instantly much better. – Gabor Angeli Oct 09 '18 at 15:04

2 Answers2

6

I have been facing this issue for a while and tried almost everything like disabling firewall,changing iptables settings.

Finally while playing around with docker-compose.yml file to set up concourse, I changed the following 2 settings in the file and resolved the issue.

CONCOURSE_GARDEN_DNS_PROXY_ENABLE=true
CONCOURSE_WORKER_GARDEN_DNS_PROXY_ENABLE=true

Hope this resolves your issue as well.

lionxlamb
  • 391
  • 5
  • 14
2

I had this same issue on an Ubuntu VM, and it turned out that my iptables chain FORWARD policy was set to DROP. Changing the FORWARD policy to ACCEPT resolved the issue. $ iptables -P FORWARD ACCEPT

Kosy Anyanwu
  • 349
  • 2
  • 6