drone.io on k8s. letsencrypt cant provide secure connection

Question

I'm deploying drone.io on gke k8s using helm. It works great if I have LetsEncrypt off. But I really would like https support.

Here's my service:

apiVersion: v1
kind: Service
metadata:
  name: {{ template "drone_ci.fullname" . }}-external
  labels:
    name: server
    app: {{ template "drone_ci.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  type: LoadBalancer
  loadBalancerIP:  {{ .Values.droneLoadBalancerIp}}
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8000
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    name: server

I have another service for port 9000 since that is only required for the drone agent.

My drone-server deployment template looks like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "drone_ci_server.fullname" . }}
  labels:
    app: {{ template "drone_ci.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: server
        app: {{ template "drone_ci.name" . }}
        release: {{ .Release.Name }}
    spec:
      containers:
      - name: server
        image: "{{ .Values.server.image.repository }}:{{ .Values.server.image.tag }}"
        imagePullPolicy: {{ .Values.server.image.pullPolicy }}
        env:
          - name: "DRONE_HOST"
            value: {{ .Values.droneHost }}
          - name: "DRONE_OPEN"
            value: "true"
          - name: "DRONE_GITLAB"
            value: "true"
          - name: DRONE_GITLAB_URL
            value: {{ .Values.droneGitlabUrl }}
          - name: DRONE_ADMIN
            value: {{ .Values.droneAdmin }}
          - name: DRONE_GITLAB_CLIENT
            valueFrom:
              secretKeyRef:
                name: {{ template "drone_ci.fullname" . }}
                key: DRONE_GITLAB_CLIENT
          - name: DRONE_GITLAB_SECRET
            valueFrom:
              secretKeyRef:
                name: {{ template "drone_ci.fullname" . }}
                key: DRONE_GITLAB_SECRET
          - name: DRONE_SECRET
            valueFrom:
              secretKeyRef:
                name: {{ template "drone_ci.fullname" . }}
                key: DRONE_SECRET
          - name: DRONE_LETS_ENCRYPT
            value: "true"
        volumeMounts:
        - mountPath: /var/lib/drone
          name: drone-lib-pv-storage
      volumes:
      - name: drone-lib-pv-storage
        persistentVolumeClaim:
          claimName: {{ template "drone_ci.fullname" . }}

When letsEncrypt is false then my site works and it connects to my gitlab instance just fine at the correct url. When letsEncrypt is true then:

Navigating to my drone in chrome gives me "This site cant provide a secure connection". ssllab't test tells me:

No secure protocols supported - if you get this message, but you know that the site supports SSL, wait until the cache expires on its own, then try again, making sure the hostname you enter uses the "www" prefix (e.g., "www.ssllabs.com", not just "ssllabs.com").
no more data allowed for version 1 certificate - the certificate is invalid; it is declared as version 1, but uses extensions, which were introduced in version 3. Browsers might ignore this problem, but our parser is strict and refuses to proceed. We'll try to find a different parser to avoid this problem.
Failed to obtain certificate and Internal Error - errors of this type will often be reported for servers that use connection rate limits or block connections in response to unusual traffic. Problems of this type are very difficult to diagnose. If you have access to the server being tested, before reporting a problem to us, please check that there is no rate limiting or IDS in place.
NetScaler issues - some NetScaler versions appear to reject SSL handshakes that do not include certain suites or handshakes that use a few suites. If the test is failing and there is a NetScaler load balancer in place, that's most likely the reason.
Unexpected failure - our tests are designed to fail when unusual results are observed. This usually happens when there are multiple TLS servers behind the same IP address. In such cases we can't provide accurate results, which is why we fail.

Looking at my pod logs, every time I try and access drone via chrome I get:

http: TLS handshake error from x.x.x.x:53938: acme/autocert: no supported challenge type found
http: TLS handshake error from y.y.y.y:53936: acme/autocert: missing certificate

My drone server image is:

image:
  repository: drone/drone
  tag: 0.8
  pullPolicy: Always

What am I missing or doing wrong?

Radek 'Goblin' Pieczonka · Accepted Answer · 2018-01-12T08:06:18.877

I would suggest you switching from a LoadBalancer type service to a regular one an instead expose it via means of Ingress. Coupling Ingress with kube-lego you get a very nice support for exposing anything you want easily with trivial way of enabling let's encrypt integration for the used domains, even if the software behind has no built in support for LE. This is in fact how my own instance of drone.io is set up.

While this might not be an answer to a root cause of your problem, which probably needs some more debug information, it's a perfectly viable and verified solution :)

As for the error it self, it seems from this code that there is no support in drone for challenges other then tls-sni-01/02 one. Among other issues that might be cluster level, there is also this issue with TLS-SNI being now disabled by LE

Sounds like a good plan. I dont know how to do that yet though. Once I have https working I'll mark this correct (unless a simler answer comes up in the meantime) — Sheena, Jan 13 '18 at 14:42

drone.io on k8s. letsencrypt cant provide secure connection

1 Answers1