Istio and Hashicorpt Vault agent Sidecar not working properly

Question

I am using on-prem k8s v1.19 and Istio with 1.8.0.. I got stuck to run them together properly when I inject istio mesh to the hub-dev where our microservices are running. Vault is running dev namespace.

The first issue I had is Vault and Istio sidecar is not running properly somehow and the application can not able to init as below. I tried to use below annotations to init first vault but it did not solve below issue.

vault.hashicorp.com/agent-init-first: true
vault.hashicorp.com/agent-inject: true

Here are the outputs of pod status and describe

$ kubectl get pods -n hub-dev
    NAME                                     READY   STATUS     RESTARTS   AGE
    oneapihub-mp-dev-59f7685455-5kmft        0/3     Init:0/2   0          19
    
$ kubectl describe pod oneapihub-mp-dev-59f7685455-5kmft -n hub-dev

Init Containers:
  vault-agent-init:
    Container ID:  
    State:          Running
      Started:      Fri, 15 Jan 2021 13:54:30 +0300
    Ready:          False
  istio-validation:
    Container ID:
    Image:         reg-dhc.app.corpintra.net/i3-mirror/docker.io_istio_proxyv2:1.8.0
    State:          Waiting
     Reason:       PodInitializing
    Ready:          False
Containers:
      oneapihub-mp:
        Container ID:
        State:          Waiting
          Reason:       PodInitializing
        Ready:          False
      istio-proxy:
        Container ID:
        State:          Waiting
          Reason:       PodInitializing
        Ready:          False
  istio-proxy:
    Container ID:
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False

    Normal  Pulled     16m   kubelet, xx-kube-node07  Container image "docker.io_vault:1.5.2" already present on machine
    Normal  Created    16m   kubelet, xx-kube-node07  Created container vault-agent-init
    Normal  Started    16m   kubelet, xx-kube-node07  Started container vault-agent-init

When I tried below annotation, it fixed the above issue, but this time when pod starts to run it can not able to find /vault/secrets path but somehow after that it can be read when I checked the logs of proxy and application and /vault/secrets folder is exist inside pod.

 - vault.hashicorp.com/agent-pre-populate: "false"

Here the logs of app even if folder exists

$ kubectl get pods -n hub-dev
oneapihub-mp-dev-78449b8cf6-qbqhn        3/3     Running   0          9m31s

$ kubectl logs -f oneapihub-mp-dev-78449b8cf6-qbqhn -n hub-dev -c oneapihub-mp

> market-place@1.0.0 start:docker /usr/src/app
> node app.js

{"message""devMessage":"SECRET_READ_ERROR","data":"","exception":"ENOENT: no such file or directory, open '/vault/secrets/database'","stack":"Error: ENOENT: no such file or directory, open '/vault/secrets/database'->

/ $ cd /vault/secrets
/vault/secrets $ ls
database  jenkins
/vault/secrets $

Here I have some PUT error which might related with Vault itself but I am confuse how then Vault can inject the secrets.

 $ kubectl logs -f oneapihub-mp-dev-78449b8cf6-qbqhn -n hub-dev -c vault-agent

2021-01-15T11:21:13.477Z [ERROR] auth.handler: error authenticating: error="Put "http://vault.dev.svc:8200/v1/auth/kubernetes/login": dial tcp 10.254.30.115:8200: connect: connection refused" backoff=2.464775515
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.5.2
             Version Sha: 685fdfa60d607bca069c09d2d52b6958a7a2febd

2021-01-15T11:21:15.942Z [INFO]  auth.handler: authenticating
2021-01-15T11:21:15.966Z [INFO]  auth.handler: authentication successful, sending token to sinks
2021-01-15T11:21:15.966Z [INFO]  sink.file: token written: path=/home/vault/.vault-token

And lastly when I checkted the istio-proxy logs I can see that GET or PUT request returns 200.

$ kubectl logs -f oneapihub-mp-dev-78449b8cf6-h8s8j -n hub-dev -c istio-proxy

021-01-15T11:35:04.352221Z  warning envoy filter    mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2021-01-15T11:35:05.557Z] "PUT /v1/auth/kubernetes/login HTTP/1.1" 200 - "-" 1294 717 8 8 "-" "Go-http-client/1.1" "a082698b-d1f7-4aa5-9db5-01d86d5093ef" "vault.dev.svc:8200" "10.6.24.55:8200" outbound|8200||vault.dev.svc.cluster.local 10.6.19.226:55974 10.254.30.115:8200 10.6.19.226:60478 - default
2021-01-15T11:35:05.724833Z info    Envoy proxy is ready
[2021-010.6.19.226:41888 - default
[2021-01-15T11:35:05.596Z] "GET /v1/secret/data/oneapihub-marketplace/database HTTP/1.1" 200 - "-" 0 400 0 0 "-" "Go-http-client/1.1" "d7d10c1f-c445-44d1-b0e3-bb9ae7bbc2f0" "vault.dev.svc:8200" "10.6.24.55:8200" outbound|8200||vault.dev.svc.cluster.local 10.6.19.226:55974 10.254.30.115:8200 10.6.19.226:41900 - default
[2021-01-15T11:35:05.591Z] "PUT /v1/auth/token/renew-self HTTP/1.1" 200 - "-" 15 717 8 8 "-" "Go-http-client/1.1" "56705e5c-c966-4bc8-8187-7ca5bb2b4abe" "vault.dev.svc:8200" "10.6.24.55:8200" outbound|8200||vault.dev.svc.cluster.local 10.6.19.226:37388 10.254.30.115:8200 10.6.19.226:41890 - default
[2021-01-15T11:35:05.602Z] "GET /v1/secret/data/oneapihub-marketplace/jenkins HTTP/1.1" 200 - "-" 0 284 0 0 "-" "Go-http-client/1.1" "1b6d8601-18df-4f32-8722-162aa785c476" "vault.dev.svc:8200" "10.6.24.55:8200" outbound|8200||vault.dev.svc.cluster.local 10.6.19.226:55974 10.254.30.115:8200 10.6.19.226:41902 - default

Your `dev` namespace, where the vault server is running, is also istio-enabled? With `vault.hashicorp.com/agent-pre-populate: "false"` you don't get the agent init container, only the side-car. That agent side-car and your application are in a race condition. So you may or may not see the secrets in your application. Maybe this also helps: https://github.com/hashicorp/vault-k8s/issues/41. Do you require Vault to be part of the mesh or would it be OK if the agent-vault-traffic is not covered by Istio? — Ralf, Jan 15 '21 at 13:04
@Ralf, No istio disabled for `dev` namespace.. Should `vault` also be meshed? Yeah with that annotation this is what ı faced thus it did not help me.. Indeed I checked github.com/hashicorp/vault-k8s/issues/41 here but really not sure which are the exact solution except the things that I applied as in my question — semural, Jan 15 '21 at 13:07
Well, without Vault in the mesh you don't get much from Istio. It mostly makes things more complicated. Either you should add Vault to the mesh or exclude the agent traffic by whitelisting port 8200 as suggested in the github thread. — Ralf, Jan 15 '21 at 13:10
@Ralf, yeah thats why I did not enable istio for Vault `vault.hashicorp.com/agent-init-first: true` why this annotation did not help me do you think ? — semural, Jan 15 '21 at 13:12
I don't know. It should. But you'd still have a problem with the side-car. I'd try excluding port 8200 as this should solve the problem for both the int container as well as the side-car. — Ralf, Jan 15 '21 at 13:17
@Ralf `traffic.sidecar.istio.io/excludeOutboundPorts: "8200"`this annotation works with the `vault.hashicorp.com/agent-init-first: true`.. Could you please help me the clarify what first annotation is doing exactly ? Do I need to create a Service Entry as well ? — semural, Jan 15 '21 at 13:37
Traffic directed to this port will be ignored by the proxy in your pod. — Ralf, Jan 15 '21 at 14:34
Is there any bad practice in security point of view ?? Do I have to create a service entry as well or this annotation is enough to work it properly. — semural, Jan 15 '21 at 14:43
Service entry for what? Depending on how your vault(-agent) setup looks like the traffic might be plaintext. This may or may not be OK, depending on your security requirements. — Ralf, Jan 15 '21 at 15:01

score 3 · Answer 1 · answered Feb 19 '21 at 12:44

3

Added below annotations worked for me.

  template:
    metadata:
      annotations:
        traffic.sidecar.istio.io/excludeOutboundPorts: "8200"
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-inject: "true"

answered Feb 19 '21 at 12:44

semural

3,583
8
37
67

score 0 · Answer 2 · answered Jul 30 '21 at 14:47

The below annotation is also working fine with me. If we use 'traffic.sidecar.istio.io/excludeOutboundPorts' which means while reading secret from the vault the traffic is not going through Istio sidecar and is most likely not encrypted.

template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-inject: "true"

score 0 · Answer 3 · answered Dec 27 '21 at 07:16

To make vault init agent work, you may need add below in your deploy:

vault.hashicorp.com/agent-pre-populate: "false" traffic.sidecar.istio.io/excludeOutboundPorts: "8200"

Finally，I think put vault out of mesh is better solution, as my document, just FYI https://github.com/johnzheng1975/devops_way/wiki/Expose-vault-with-internal-domain

Istio and Hashicorpt Vault agent Sidecar not working properly

3 Answers3