I've been following this tutorial to set up vault and kubernetes on minikube with helm.
It seems to me the vault service account is using the default service account JWT token to access the API to authenticate. Should the vault service account instead be using its own token, not the default one?
Here are the relevant configuration steps:
# Configure the auth.
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
disable_iss_validation=true
# Configure the policy.
vault policy write webapp - <<EOF
path "secret/data/webapp/config" {
capabilities = ["read"]
}
EOF
# Create the role.
vault write auth/kubernetes/role/webapp \
bound_service_account_names=vault \
bound_service_account_namespaces=default \
policies=webapp \
ttl=24h
The client (expressjs) on the webapp pod reads the mounted token to authenticate:
...
await axiosInst.post("/auth/kubernetes/login",
{jwt: fs.readFileSync(process.env.JWT_PATH, {encoding: "utf-8"}),
role: "webapp"});
...
However, the token at /var/run/secrets/kubernetes.io/serviceaccount/token on the webapp pod is the default service account token, not the vault service account token right?
Am I doing something wrong here?
web-deploy.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 2
selector:
matchLabels:
app: web-pod
template:
metadata:
labels:
app: web-pod
spec:
serviceAccountName: vault
containers:
- name: web
image: kahunacohen/hello-k8s
imagePullPolicy: IfNotPresent
env:
- name: VAULT_ADDR
value: 'http://vault:8200'
- name: JWT_PATH
value: '/var/run/secrets/kubernetes.io/serviceaccount/token'
- name: SERVICE_PORT
value: '8080'
envFrom:
- configMapRef:
name: web-configmap
ports:
- containerPort: 3000
protocol: TCP
I describe the vault sa:
$ kubectl describe serviceaccounts vault
Name: vault
Namespace: default
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=vault
helm.sh/chart=vault-0.14.0
Annotations: meta.helm.sh/release-name: vault
meta.helm.sh/release-namespace: default
Image pull secrets: <none>
Mountable secrets: vault-token-jhp5q
Tokens: vault-token-jhp5q
Events: <none>
I describe the secret associated with this sa:
$ kubectl describe secret vault-token-jhp5q
Name: vault-token-jhp5q
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name: vault
kubernetes.io/service-account.uid: 9a091a35-9292-4d77-abac-a7d62f312d27
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1111 bytes
namespace: 7 bytes
token: eyJhb...
But when I cat the token on my pod it doesn't match above:
$ kubectl exec web-deployment-66968775cb-8td6c -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
...
pod yaml:
$ kubectl get pods web-deployment-66968775cb-8td6c -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2021-08-14T19:32:54Z"
generateName: web-deployment-66968775cb-
labels:
app: web-pod
app.kubernetes.io/managed-by: skaffold
pod-template-hash: 66968775cb
skaffold.dev/run-id: f58f771b-258a-4304-baf6-1cc4d9bc3029
name: web-deployment-66968775cb-8td6c
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: web-deployment-66968775cb
uid: 6e62ba08-8b56-402a-ae10-e5d95f93dc39
resourceVersion: "1444983"
uid: 991199f5-0e53-45b9-a4d7-ab1d3cc44079
spec:
containers:
- env:
- name: VAULT_ADDR
value: http://vault:8200
- name: JWT_PATH
value: /var/run/secrets/kubernetes.io/serviceaccount/token
- name: SERVICE_PORT
value: "8080"
envFrom:
- configMapRef:
name: web-configmap
image: kahunacohen/hello-k8s:fb961e142e46220f11853eba6f50eaf94510a95dc08046ad8a1517d92fcb2d85
imagePullPolicy: IfNotPresent
name: web
ports:
- containerPort: 3000
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zjkvp
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: minikube
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: vault
serviceAccountName: vault
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-zjkvp
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2021-08-14T19:32:54Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2021-08-15T15:01:02Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2021-08-15T15:01:02Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2021-08-14T19:32:54Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://4759183417a8ef0bbee8f28bf345aed373371f7ca462b9a5f5a966eba33117d2
image: kahunacohen/hello-k8s:b65b899-dirty
imageID: docker://sha256:fb961e142e46220f11853eba6f50eaf94510a95dc08046ad8a1517d92fcb2d85
lastState:
terminated:
containerID: docker://87636058668d022160149f176cd3c565bf17c239e97c55d873e42c9c074215d6
exitCode: 255
finishedAt: "2021-08-15T15:00:01Z"
reason: Error
startedAt: "2021-08-14T19:32:55Z"
name: web
ready: true
restartCount: 1
started: true
state:
running:
startedAt: "2021-08-15T15:01:01Z"
hostIP: 192.168.49.2
phase: Running
podIP: 172.17.0.10
podIPs:
- ip: 172.17.0.10
qosClass: BestEffort
startTime: "2021-08-14T19:32:54Z"
