Pass json string to environment variable in a k8s deployment for Envoy

Question

I have a K8s deployment with one pod running among others a container with Envoy sw. I have defined image in such way that if there is an Environment variable EXTRA_OPTS defined it will be appended to the command line to start Envoy. I want to use that variable to override default configuration as explained in https://www.envoyproxy.io/docs/envoy/latest/operations/cli#cmdoption-config-yaml

Environment variable works ok for other command options such as "-l debug" for example. Also, I have tested expected final command line and it works.

Dockerfile set Envoy to run in this way:

CMD ["/bin/bash", "-c", "envoy -c envoy.yaml $EXTRA_OPTS"]

What I want is to set this:

       ...
      - image: envoy-proxy:1.10.0
        imagePullPolicy: IfNotPresent
        name: envoy-proxy
        env:
           - name: EXTRA_OPTS
             value: ' --config-yaml "admin: { address: {  socket_address: { address: 0.0.0.0, port_value: 9902 } } }"'
        ...

I have succesfully tested running envoy with final command line:

envoy -c /etc/envoy/envoy.yaml --config-yaml "admin: { address: {  socket_address: { address: 0.0.0.0, port_value: 9902 } } }"

And I have also tested a "simpler" option in EXTRA_OPTS and it works:

       ...
      - image: envoy-proxy:1.10.0
        imagePullPolicy: IfNotPresent
        name: envoy-proxy
        env:
           - name: EXTRA_OPTS
             value: ' -l debug'
        ...

I would expect Envoy running with this new admin port, instead I'm having param errors:

PARSE ERROR: Argument: {
             Couldn't find match for argument

It looks like quotes are not being passed to the actual Environment variable into the container...

Any clue???

Thanks to all

Answer 1

You should set ["/bin/bash", "-c", "envoy -c envoy.yaml"] as an ENTRYPOINT in you dockerfile or use command in kubernetes and then use args to add additional arguments.

You can find more information in docker documentation

Let me explain by example:

$ docker build -t fl3sh/test:bash .

$ cat Dockerfile
FROM ubuntu
RUN echo '#!/bin/bash' > args.sh && \
    echo 'echo "$@"' >> args.sh && \
    chmod -x args.sh
CMD ["args","from","docker","cmd"]
ENTRYPOINT ["/bin/bash", "args.sh", "$ENV_ARG"]

cat args.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: args
  name: args
spec:
  containers:
  - args:
    - args
    - from
    - k8s
    image: fl3sh/test:bash
    name: args
    imagePullPolicy: Always
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

Output:
pod/args $ENV_ARG args from k8s

cat command-args.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: command-args
  name: command-args
spec:
  containers:
  - command:
    - /bin/bash
    - -c
    args:
    - 'echo args'
    image: fl3sh/test:bash
    imagePullPolicy: Always
    name: args
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

Output:
pod/command-args args

cat command-env-args.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: command-env-args
  name: command-env-args
spec:
  containers:
  - env:
    - name: ENV_ARG
      value: "arg from env"
    command:
    - /bin/bash
    - -c
    - exec echo "$ENV_ARG"
    image: fl3sh/test:bash
    imagePullPolicy: Always
    name: args
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

Output:
pod/command-env-args arg from env

cat command-no-args.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: command-no-args
  name: command-no-args
spec:
  containers:
  - command:
    - /bin/bash
    - -c
    - 'echo "no args";echo "$@"'
    image: fl3sh/test:bash
    name: args
    imagePullPolicy: Always
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

Output:
pod/command-no-args no args

#notice ^ empty line above  

cat no-args.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: no-args
  name: no-args
spec:
  containers:
  - image: fl3sh/test:bash
    name: no-args
    imagePullPolicy: Always
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

Output:
pod/no-args $ENV_ARG args from docker cmd

If you need to recreate my example you can use this loop to get this output like above:

for p in `kubectl get po -oname`; do echo cat ${p#*/}.yaml; echo ""; \
cat ${p#*/}.yaml; echo -e "\nOutput:"; printf "$p "; \
kubectl logs $p;echo "";done

Conclusion if you need to pass env as arguments use:

    command:
    - /bin/bash
    - -c
    - exec echo "$ENV_ARG"

I hope now it is clear.