Using kubernetes-dashboard with outh2-proxy and keycloak (and EKS) - unauthorized

Question

I am trying to use oauth-proxy to provide authentication on the kubernetes dashboard using keycloak in EKS.

I have managed to get to a point where oauth-proxy will forward the authorization header to the dashboard, however I am getting 'unauthorized' in the dashboard.

From what I can gather, the dashboard is expecting a field in the header with the id_token however the version of keycloak I am using does not seem to provide it... It is not clear to me if this is an issue with the way I have configured keycloak or an issue with something else.

Both oauth-proxy and kubernetes dashboard are deployed using helm with the following config:

kubernetes-dashboard:
  app:
    ingress:
      enabled: true 
      ingressClassName: nginx
      issuer:
        name: letsencrypt
        scope: cluster
      paths:
        web: /
        api: /api
      annotations: 
        external-dns.alpha.kubernetes.io/hostname: kubernetes-dashboard.example.com
        nginx.ingress.kubernetes.io/proxy-buffer-size: "64k"
        nginx.ingress.kubernetes.io/backend-protocol: HTTP
        nginx.ingress.kubernetes.io/auth-signin: 'https://kubernetes-dashboard.example.com/oauth2/start?rd=$escaped_request_uri'
        nginx.ingress.kubernetes.io/auth-url: 'https://kubernetes-dashboard.example.com/oauth2/auth'
        nginx.ingress.kubernetes.io/auth-response-headers: "Authorization"
      hosts:
        - kubernetes-dashboard.example.com

  nginx:
    enabled: false

  cert-manager:
    enabled: false
    installCRDs: false
  
  metrics-server:
    enabled: false

oauth2-proxy: 
  config:
    existingSecret: "kubernetes-dashboard-oidc-secret"
    configFile: | 
      provider="keycloak-oidc"
      provider_display_name="Keycloak"
      redirect_url="https://kubernetes-dashboard.example.com/oauth2/callback"

      email_domains = [ "*" ]
      oidc_issuer_url="https://keycloak.example.com/realms/myrealm"
      scope = "openid email groups"
      upstreams = [ "https://kubernetes-dashboard.example.com" ]

      cookie_secure = true

      set_authorization_header = true

  metrics:
    enabled: false

  ingress:
    enabled: true
    path: /oauth2
    className: nginx
    annotations: 
      nginx.ingress.kubernetes.io/proxy-buffer-size: "64k"
    hosts:
      - kubernetes-dashboard.example.com
    tls:
      - secretName: kubernetes-dashboard-tls
        hosts:
          - "kubernetes-dashboard.example.com"

  sessionStorage:
    type: redis
    redis:
      password: ""

  redis: 
    enabled: true 
    architecture: standalone

Versions

oauth2-proxy: Helm 6.16.1 (app 7.4.0)

kubernetes-dashboard: Helm 7.0.3 (app v3.0.0-alpha0)

Keycloak: 21.1.1

EKS Version: 1.26

EKS OIDC Config

Keycloak Config

I am using the same client in keycloak that I use for kubelogin using oidc (which does work).

EKS Logs*

I believe these are the errors for the dashboard - but sadly they are not clear...

jwt token

So when I visit https://kubernetes-dashboard.example.com/oauth2/callback I can see the authorization header but it does not contain an id_token...

When I get to the page - it logs in, however none of the resources are displayed... I am assuming that this is because the Authorization token is being passed and it does not contain the id_token, however nothing I have seemed to try appears to get it into the header... This would confirm why I am getting the errors in the API server...

Can anyone help?

I'm curious, are you able to run kubectl commands against the cluster after you authenticate to Keycloak? — Jeremy Cowan, Aug 17 '23 at 14:52
Hey @JeremyCowan - yes I can and that works fine... I have used this https://github.com/int128/kubelogin to do this, but I have a feeling that kubectl does an intermediate step to get the actual token. — geoffo-dev, Aug 18 '23 at 10:59
I haven't seen this particular configuration before, i.e. using Keycloak to authenticate to the k8s dashboard, but I found an old blog post that may help, https://www.tigera.io/blog/single-sign-on-for-kubernetes-dashboard-experience/. Incidentally, you can view all the objects in a cluster from the EKS console. As an alternative the dashboard, you can use Octant, k9s, or Lens. — Jeremy Cowan, Aug 18 '23 at 14:16
So there are a couple of blog posts out there that talk about it - I think as long as the token is in there, it should pass this through to the api and this is what it would use to pull through the details... Looking into it, it seems like this is an issue with the oauth2-proxy and that using the `alphaConfig` option I should be able to inject the token into the response - https://github.com/oauth2-proxy/oauth2-proxy/issues/843. So I will be honest I am not a massive fan of the EKS dashboard (and requires console access). Currently using k9s and will have a look through the links! Will update! — geoffo-dev, Aug 18 '23 at 14:41
So it does look like the id_token is being used as a bearer token (or at least passed), although I am still getting 'unauthorized' everywhere... — geoffo-dev, Aug 18 '23 at 17:14
are you able to login to the dashboard when you paste in the token that keycloak returns? — Jeremy Cowan, Aug 18 '23 at 17:27

Gary Archer · Answer 1 · 2023-08-19T08:07:06.977

You are correctly supplying a token with the required groups claim, so users can authenticate. But the K8S API server needs to know what K8S resources to grant access to, so users are unauthorized.

So you next need to deploy RBAC resources for each group. For example, apply this YAML to grant read access to all K8S resources to members of the admins group:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: dashboard-admins-role
rules:
- apiGroups: ['*']
  resources: ['*']
  verbs: ['get', 'list', 'watch']
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dashboard-admins-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dashboard-admins-role
subjects:
- kind: Group
  name: admins

Sorry for the delayed reply! Havent had a chance to revisit this! I did have these groups in place, but I have gone back to make sure the oidc-login is working and it isn't... looks like the configuration has changed... will get this working and then try again! Thanks for the steer! — geoffo-dev, Aug 23 '23 at 11:07

score 0 · Accepted Answer · answered Aug 23 '23 at 11:58

so it turns out I had made an error in the OIDC configuration for EKS... As I am now using a later version of keycloak, I needed to make sure the issuer url was:

https://keycloak.example.com/realms/myrealm

and not

https://keycloak.example.com/auth/realms/myrealm

Frustratingly I had changed it everywhere else and not there... Everything now works as expected!

Thank you!!

Using kubernetes-dashboard with outh2-proxy and keycloak (and EKS) - unauthorized

2 Answers2