gcloud beta pubsub subscriptions pull format

Question

Quick question: When I pull message from pubsub subscription via command line tool

gcloud beta pubsub subscriptions pull MY_SUB

I am getting a table with (all details and) data as string (already decoded) But i want to use it so i did:

gcloud beta pubsub subscriptions pull MY_SUB --format=json

Than i receive a json with (all details) but the data is encoded.

There is a way to get it parsed with formatting?

Example of publishing message:

gcloud beta  pubsub topics publish myTopic "Topic Message" --attribute=Ai=A,Bee=B

NO-FORMATTING_RETURN

  ─────────────┬─────────────────┬────────────────┬─────────────────────────

  ──────────────────────────────────────────────────────────────────────────
    ─────────────────────────────────────────────────────────────┐
    │     DATA    │    MESSAGE_ID   │   ATTRIBUTES   │                                                                             
    ACK_ID                                                                             
    │

  ├─────────────┼─────────────────┼────────────────┼────────────────────────

  ──────────────────────────────────────────────────────────────────────────
     ──────────────────────────────────────────────────────────────┤
     │ Topic Message │ 122122177601805 │ Ai=A Bee=B  │ ACK_ID... │

  └─────────────┴─────────────────┴────────────────┴────────────────────────

  ──────────────────────────────────────────────────────────────────────────
    ──────────────────────────────────────────────────────────────┘

FORMATTING

[
{
"ackId": "ACK_ID..",
"message": {
  "attributes": {
    "Ai": "A",
      "Bee": "B"
    },
     "data": "SGVsbG8gVG9waWM=",
     "messageId": "122121955409996",
     "publishTime": "2017-05-11T10:26:54.143Z"
    }
}
]

Technetium · Accepted Answer · 2021-08-02T18:18:47.430

11

You are on the right track with the use of the --format argument, but you need to use projections in order to decode the data. In this case, you need to use the decode() projection. Here's how you can perform the same command with the same data except the message's data is base64 decoded.

gcloud beta pubsub subscriptions pull MY_SUB --format="json(ackId, message.attributes, message.data.decode(\"base64\").decode(\"utf-8\"), message.messageId, message.publishTime)"

[
  {
    "ackId": "QV5AEkw...D5-NTlF",
    "message": {
      "attributes": {
        "Ai": "A",
        "Bee": "B"
      },
      "data": "Topic Message",
      "messageId": "127236468931635",
      "publishTime": "2017-05-29T23:15:04.637Z"
    }
  }
]

edited Aug 02 '21 at 18:18

answered May 29 '17 at 23:20

Technetium

5,902
2
43
54

For decoded message.data in JSON format, how do you eliminate the backslash-quote \" escaping in the output? – James May 21 '18 at 19:16
That sounds like a new question :) There appears to be a `sub()` transform you could use in the `--format` argument after the `json()` projection, or you could always pipe it into something like `sed`. – Technetium May 22 '18 at 01:04
Running this command now may result in an error `ERROR: gcloud crashed (TypeError): Object of type bytes is not JSON serializable` This can be corrected with an additional the projection decoding `--format="json(message.attributes, message.data.decode(\"base64\").decode(\"utf- 8\"), message.messageId, message.publishTime)" ` – Scott Aug 02 '21 at 17:59

score 0 · Answer 2 · answered Aug 28 '23 at 20:49

This worked for my use case:

gcloud pubsub subscriptions pull test-sub --format="table[no-heading](DATA)" > alert.json

When I tried the above examples I kept getting errors:

base64: invalid input

jq: error (at pull.json:10): string ("ewogICJpbm...) is not valid base64 data

gcloud beta pubsub subscriptions pull format

2 Answers2