0

I have a deployment that needs the ip address of the cloud redis instance.

I'm creating the cloud redis instance via config connector:

apiVersion: redis.cnrm.cloud.google.com/v1beta1
kind: RedisInstance
metadata:
  name: redis-name
  annotations:
    cnrm.cloud.google.com/project-id: project-id
spec:
  region: region
  displayName: Cloud Redis
  tier: BASIC
  memorySizeGb: 1
  authorizedNetworkRef:
    external: projects/project-id/global/networks/network-name

I have a deployment where I want to add this via an env var

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-name
spec:
  template:
    spec: 
      containers:
      - name: web
        env:
        - name: REDIS_HOST
          value: "needs to be replaced"

I have tried to replace it a few ways with no success

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: my-namespace
bases:
  - ../../base 

replacements:
  - source:
      kind: RedisInstance 
      name: redis-name
      namespace: my-namespace
      fieldPath: status.host

I get the error

fieldPath `status.host` is missing for replacement source RedisInstance.[noVer].[noGrp]/redis-name.my-namespace

I've also tried with

vars:
 - name: REDIS_HOST
   objref:
     kind: RedisInstance
     name: redis-name
     apiVersion: redis.cnrm.cloud.google.com/v1beta1
   fieldref:
     fieldpath: status.host

I'm assuming that this can't be done because status doesnt exist until the resource is "live".. Is there a better way to do this?

In Terraform I would be able to reference the existing resource.. Seems like in Kustomize this isn't possible?

ZiggyStardust
  • 33
  • 5
  • Kustomize doesn't interact with the remote cluster -- it only knows about things defined in the source manifests listed in your `resources` section (modified by patches, etc). – larsks Jun 07 '23 at 00:01

2 Answers2

1

Here is an another way to reference the redis host/ip created by config connector in GKE from a Kustomization

1.Create a customization with REDIS_HOST env

# cat overlays/test/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
bases:
  - ../../base/

replacements:
  - source:
      kind: RedisInstance
      name: redis-test
      namespace: default
      fieldPath: status.host
    targets:
    - select:
        name: hello
      fieldPaths:
      - spec.template.spec.containers.[name=hello].env.[name=REDIS_HOST].value

Note: above yaml only works when cat base/redis.yaml have status.host.

2.This is redis instance have below yaml:

spec:
    authorizedNetworkRef:
      external: projects/gkeprivate/global/networks/default
    connectMode: DIRECT_PEERING
    displayName: Cloud Redis
    locationId: us-central1-b
    memorySizeGb: 1
    persistenceConfig:
      persistenceMode: DISABLED
    readReplicasMode: READ_REPLICAS_DISABLED
    redisVersion: REDIS_6_X
    region: us-central1
    resourceID: redis-test
    tier: BASIC
    transitEncryptionMode: DISABLED
  status:
    conditions:
    - lastTransitionTime: "2023-06-28T10:57:44Z"
      message: The resource is up to date
      reason: UpToDate
      status: "True"
      type: Ready
    createTime: "2023-06-28T10:53:23.758110949Z"
    currentLocationId: us-central1-b
    host: 10.34.137.179
    nodes:
    - id: node-0
      zone: us-central1-b

Your yaml is failing because originally the RedisInstance manifest doesn't have status.host nor you can define it while creating the RedisInstance manifest.

3.Deployment file :

# cat base/deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app: hello
spec:
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: hello
          image: ubuntu
          env:
            - name: REDIS_HOST
              value: "needs to be replaced"
          command:
            - sleep
            - infinity

4.At last I can see REDIS_HOST env inside POD

root@hello-cc98748c4-kfw99:/# env | grep RED
REDIS_HOST=10.34.137.17

I don't see any way through you can supply REDIS_HOST env unless you create a redis resource and then update redis.yaml (kubectl get redisinstance -o yaml > ../../base/redis.yaml) then only it looks possible to supply as env

Sai Chandra Gadde
  • 2,242
  • 1
  • 3
  • 15
  • So is it possible to reference the redis kubernetes object from the configmap? The google example linked just shows hard coding an ip address. ``` export REDISHOST_IP=XXX.XXX.XXX.XXX kubectl create configmap redishost --from-literal=REDISHOST=${REDISHOST_IP} ``` This doesn't seem any better than hard coding the ip address into the kustomization file? – ZiggyStardust Jun 07 '23 at 17:42
  • yes, the process is similar to hardcoding the IP in the config file. Though I have not came across any solution to overcome the issue yet, I am continuing the analysis to see if there is a workaround. – Sai Chandra Gadde Jun 09 '23 at 12:27
  • @ZiggyStardust I have edited the answer, i found another method and added with yaml file. – Sai Chandra Gadde Jun 29 '23 at 09:09
  • 1
    Nice find. I am able to use the status:host path if I add it to the RedisInstance yaml, however the deployment gets set to whatever is in my Redis yaml, not the live value from the redisinstance. If my redis yaml has status: tbd then the pod will get REDIS_HOST=tbd, and the redis instance is perpetually out of sync because the live / desired values differ. I'm convinced this isn't possible today with ArgoCD. – ZiggyStardust Jun 30 '23 at 14:50
1

I recently had to do something like this. You can create a PostSync Job that runs kubectl get redisinstance <your redisinstance name> -o json | jq -r .status.host and inject it into your live deployment manifest via a configmap. After all the Redis instance is a K8s resource in Config Connector. The Job will need to be run with a K8s service account bound to a GCP service account via Workload Identity and given get permission for redisinstances in the APIGroup redis.cnrm.cloud.google.com and create and update permission for configmaps.

tnqz
  • 26
  • 2