Secret is not decoding properly using Kubernetes Secrets

Question

I am using Kubernetes to deploy my grafana dashboard and I am trying to use Kubernetes Secrets for saving grafana admin-password .. Here is my yaml file for secret

    apiVersion: v1
    kind: Secret
    metadata:
      name: $APP_INSTANCE_NAME-grafana
      labels:
        app.kubernetes.io/name: $APP_INSTANCE_NAME
        app.kubernetes.io/component: grafana
    type: Opaque
    data:
      # By default, admin-user is set to `admin`
      admin-user: YWRtaW4=
      admin-password: "$GRAFANA_GENERATED_PASSWORD"

value for GRAFANA_GENERATED_PASSWORD is base64 encoded and exported like

export GRAFANA_GENERATED_PASSWORD="$(echo -n $PASSWORD | base64)"

where PASSWORD is a variable which i exported on my machine like export PASSWORD=qwerty123

I am trying to pass the value of GRAFANA_GENERATED_PASSWORD to the yaml file for secret like

envsubst '$GRAFANA_GENERATED_PASSWORD'  > "grafana_secret.yaml"

The yaml file after passing the base64 encoded value looks like

apiVersion: v1
kind: Secret
metadata:
  name: kafka-monitor-grafana
  labels:
    app.kubernetes.io/name: kafka-monitor
    app.kubernetes.io/component: grafana
type: Opaque
data:
  # By default, admin-user is set to `admin`
  admin-user: YWRtaW4=
  admin-password: "cXdlcnR5MTIz"

After deploying all my objects i couldn't login to my dashboard using password qwerty123 which is encoded properly ..

But when i try to encode my password like export GRAFANA_GENERATED_PASSWORD="$(echo -n 'qwerty123' | base64)"

It is working properly and i can login to my dashboard using the password qwerty123 .. Looks like the problem occur when i encode my password using a variable ... But i have encode my password using a variable

@KoopaKiller yes ,and i tried to decode the base64 result from both methods .. its decoding to my original password ... — Pratheesh, May 04 '20 at 11:34
I've test your commands here and both way to generate the password is working fine... the only issue I found is in the envsubst command, it just workus if I pass the original file in the command line, example: `envsubst '$GRAFANA_GENERATED_PASSWORD' "modified_code.yaml"` in the way you have posted in your example didn't worked for me. Are you using some automation to do it? Try to check if the file is been generating correctly — Mr.KoopaKiller, May 04 '20 at 13:08
@KoopaKiller sorry i am combining some file to form one single master file like ..awk 'FNR==1 {print "---"}{print}' manifest/* | envsubst '$APP_INSTANCE_NAME $NAMESPACE $GRAFANA_GENERATED_PASSWORD' > "${APP_INSTANCE_NAME}_manifest.yaml . — Pratheesh, May 04 '20 at 14:08
Just for clarify: this behavior occurs when you try to change the password and re-applying the secret or for a new deployment? I'm guessing you are following [this](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-kubernetes-monitoring-stack-with-prometheus-grafana-and-alertmanager-on-digitalocean) guide to setup your environment, I've tested in both ways and it works for me. — Mr.KoopaKiller, May 05 '20 at 08:42
Additionnaly, you could pass the same base64 encode to `admin-user` in your yaml and check the grafana pod log's, there is an especific line that mention the name of the admin user that will be created. Maybe some junk chars is hidden in the encode string (\n). — Mr.KoopaKiller, May 06 '20 at 07:12
Hi @KoopaKiller i am using the same doc u have mentioned . i am facing this issue when i try to change the password and re-applying the secret or for a new deployment.. — Pratheesh, May 06 '20 at 15:30
Hi @KoopaKiller i found the issue .. The pvc for grafana was retained even after removal of pods ..we can solve this issue by forcefully removing the pvc after removing all pods and re-apply with new secret — Pratheesh, May 07 '20 at 05:14
I was wondering about the PVC, and I got the same result as you. But, after remove the PVC and volmeMounts from the generated template, for my surprise I was unable to login in the grafana dashboard using the password and the old one. Even after recreate the grafana pod and confirm the pod got the correct secrets, I was still unabled to login in grafana dashboard. To make it works I was need to delete all the stack and recreate all again, and I can't find why since the db isn't in a volume anymore. But it is another issue. Happy to now you solve. How about post a answer to help other? — Mr.KoopaKiller, May 07 '20 at 06:47
Hi @KoopaKiller , I think if you delete the pv and and pvc you need to recreate the stack or recreate the pv and pvc for the grafana dashboard.Im not sure this is the right way of approach — Pratheesh, May 08 '20 at 05:18
Hi Pratheesh, this is the point... I'm not using PV, PVC and any other volume in the deployment... I don't know why but after changed and updated the secret, I'm unable to login in the grafana dashboard even using the old password. But it's another issue not related with your question =) — Mr.KoopaKiller, May 08 '20 at 08:28

score 1 · Accepted Answer · answered May 08 '20 at 08:36

As mentioned in @Pratheesh comment, after deploy the grafana for the first time, the persistent volume was not deleted/recreated and the file grafana.db that contains the Grafana dashboard password still keeping the old password.

In order to solve, the PersistentVolume (pv) need to be deleted before apply the secret with the new password.

Secret is not decoding properly using Kubernetes Secrets

1 Answers1