Serving ArgoCD behind Traefik in Kubernetes

Question

I would like to run ArgoCD behind Traefik such that I can both access the ArgoCD web ui and interact with the API using the argocd command line tool.

• I have deployed ArgoCD into a Kubernetes cluster from the upstream manifests.

• I have deployed Traefik as in ingress server from the helm chart. I haven't set any chart values other than the log level.

• I am using the following Ingress resource:

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: argocd
  
  spec:
    tls:
      - hosts:
        - argocd.internal
        secretName: argocd-certificate
    rules:
      - host: argocd.internal
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: argocd-server
                  port:
                    name: http
  

• The secret argocd-certificate exists and has a valid certificate and key in tls.crt and tls.key.

With this in place, I can access ArgoCD at http://argocd.internal, but attempts to access it at https://argocd.internal fail with:

$ curl https://argocd.internal
404 page not found

(Note that there is no certificate error; the endpoint is servering the certificate configured in the Ingress resource.)

I'm not sure what's going on here: I haven't bound this to any particular Traefik endpoint, so I would expect it to be available on all endpoints...and when we query the Traefik API, we see:

$ curl -sf 'http://localhost:9000/api/http/routers?search=&status=&per_page=6&page=1' |
  jq '.[]|select(.service == "argocd-argocd-server-80")'
{
  "entryPoints": [
    "metrics",
    "web",
    "websecure"
  ],
  "service": "argocd-argocd-server-80",
  "rule": "Host(`argocd.internal`) && PathPrefix(`/`)",
  "status": "enabled",
  "using": [
    "metrics",
    "web",
    "websecure"
  ],
  "name": "argocd-argocd-argocd-internal@kubernetes",
  "provider": "kubernetes"
}

Which confirms that the router is available on all endpoints.

What's going on here -- why does this only seem to do the right thing when accessed via http:// and not via https://?

If it helps, the argocd-server service looks like:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
  name: argocd-server
  namespace: argocd
spec:
  clusterIP: 10.96.117.77
  clusterIPs:
  - 10.96.117.77
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: https
    port: 443
    targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server

Answer 1

I was able to get things working with a pair of IngressRoute resources:

1. One for the http:// endpoint:

   
   apiVersion: traefik.containo.us/v1alpha1
   kind: IngressRoute
   metadata:
     name: argocd-http
   spec:
     entryPoints:
       - web
     routes:
     - kind: Rule
       match: Host(`argocd.internal`)
       priority: 10
       middlewares:
         - name: redirect-http-https
       services:
       - kind: Service
         name: argocd-server
         port: http
   

   This uses a redirecting middleware to redirect all requests to the https:// endpoint:

   apiVersion: traefik.containo.us/v1alpha1
   kind: Middleware
   metadata:
     name: redirect-http-https
   spec:
     redirectScheme:
       scheme: https
       permanent: true
   

   That means the services: portion of the IngressRoute is effectively a no-op, but it's required by the schema.

2. And one for the https:// endpoint:

   apiVersion: traefik.containo.us/v1alpha1
   kind: IngressRoute
   metadata:
     name: argocd-https
   spec:
     entryPoints:
       - websecure
     routes:
     - kind: Rule
       match: Host(`argocd.internal`)
       priority: 10
       services:
       - kind: Service
         name: argocd-server
         port: http
     - kind: Rule
       match: Host(`argocd.internal`) && Headers(`Content-Type`, `application/grpc`)
       priority: 11
       services:
       - kind: Service
         name: argocd-server
         port: http
         scheme: h2c
     tls:
       secretName: argocd-certificate
   

   We use the Content-Type to differentiate gRPC requests from HTTP requests and use the appropriate protocol scheme for the connection to the backend server.

Answer 2

Verify that your service itself works. Traefik may work correctly, but the service may be unavailable due to failed health checks, mismatched labels or security policies.

Try accessing it with dig, ping, and telnet from within Kubernetes, e.g. with the netshoot container image.

Use the internal service name as a hostname:

<name>.<namespace>.svc.cluster.local