0

After installing microk8s and then enabling kubeflow I'm given the username, password and link to Kubeflow dashboard. Then I access the dashboard as expected and all is well. BUT after restarting my machine and executing microk8s start I can no longer get to the kubeflow dashboard.

All the pods start fine and then I go to access the dashboard and get:

Access to 10.64.140.44.nip.io was denied
You don't have authorisation to view this page.
HTTP ERROR 403

Looking at the kubernetes logs for the pod/container oidc-gatekeeper-xxxxx / oidc-gatekeeper I have:

level=error msg="Failed to exchange authorization code with token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Invalid client credentials.\"}" ip=10.1.252.88 request="/authservice/oidc/callback?code=ipcb55gymqsy5pcgjn7eaenad&state=MTYyMjYzNjE4OHxFd3dBRURoMVZtSm9Wak4yUXpWQlYxZ3pPVWs9fPTKezGok06ig6bjtYvWt9sqhzaCpO_xhSMeTUFDL81j"

And for pod/container dex-auth-5d9bf87db9-rjtm8 / dex-auth:

level=info msg="invalid client_secret on token request for client: authservice-oidc"

Only by removing microk8s altogether and reinstalling everytime I restart my machine can I get this working again which is obviously not workable.

Any help would be greatly appreciated.

Nathan
  • 37
  • 4

2 Answers2

2

I've managed to resolve the issue but I'm not 100% sure which action resolved it.

I tried using Firefox rather than Chrome and noticed some documentation used IP http://10.64.140.43.nip.io/ rather than http://10.64.140.44.nip.io/.

Having been refused access as above for http://10.64.140.44.nip.io/ I found http://10.64.140.43.nip.io/ took me straight into the dashboard.

I restarted my machine to see if it was just the IP (note: checking "microk8s kubectl get services -n kubeflow" specified 10.64.150.44 as the external IP), but this time http://10.64.140.44.nip.io/ just gave me the dex log in screen and after logging in took me to the dashboard without issue.

Perhaps I just did something wrong somewhere, I'm not sure and can't check now it works as it should. Apologise if you get here with the issue and this doesn't help.

Nathan
  • 37
  • 4
0

I had a similar error. Solution for me was to enable dns, istio, and storage first. Wait until the pods were running, and then enable Kubeflow. Then make sure to port-forward using the istio-system namespace with the istio-ingressgateway pod. Kubeflow also makes a istio-ingressgateway pod, but connecting to that yielded the error. Per Kubeflow guide

mr_n0b0dy
  • 33
  • 1
  • 4