Storing binary data in Google Secret Manager

Question

I'm using Google Secret Manager for the first time to store some binary data. When I access the secret, it seems to have a different encoding or format.

The actual data is a certificate bundle, but I've been able to reproduce the issue using smaller binary data. Steps to reproduce:

1. Create a file with binary data:

   echo -e -n '\xed\xfe' > secret.txt
   

2. Create the secret and version:

   gcloud secrets create "my-secret" \
     --data-file ./secret.txt \
     --replication-policy "automatic"
   

3. Access the secret and save the result to a file:

   gcloud secrets versions access latest --secret "my-secret" > result.txt
   

4. Compare the two files:

   od -t x1 secret.txt  # ed fe
   od -t x1 result.txt  # 3f 3f 0a
   

Why is the result different? Do I have to do something extra to get Google Secret Manager to work with binary data?

Answer 1

Secret Manager stores data exactly as given. Unfortunately there was a bug in the gcloud CLI tool that was adding an additional newline character to the end of a response.

This bug was fixed in gcloud v288.0.0. Please make sure you are using v288.0.0 or higher.

If you're concerned about local encoding issues, you should acquire the raw JSON response instead. This response will include the base64-encoded secret payload, which is much safer for transport:

gcloud secrets versions access latest --secret "my-secret" --format "json"

You can use a tool like jq to parse the JSON on the command line. Note, the secret payload data is base64-encoded, so you will need to decode the value prior to using it.

gcloud secrets versions access latest --secret "my-secret" --format "json" | \
  jq -r .payload.data | \
  base64 --decode > results_binary.txt

Verify:

od -t x1 results_binary.txt  # ed fe