1

I started playing with Google Cloud IOT Core today. I went through the tutorial and ran a script to send messages from the virtual device back to my topic (if this is the correct phrasing). If I want to pull a message sent by my virtual device, I run the following:

gcloud pubsub subscriptions pull --auto-ack projects/glass-potion-314513/subscriptions/my-subscription

And the following is returned:

┌──────────────────────────────────────┬──────────────────┬──────────────┬────────────────────────────────────┬──────────────────┐
│                 DATA                 │    MESSAGE_ID    │ ORDERING_KEY │             ATTRIBUTES             │ DELIVERY_ATTEMPT │
├──────────────────────────────────────┼──────────────────┼──────────────┼────────────────────────────────────┼──────────────────┤
│ my-registry/my-node-device-payload-3 │ 2426853517884309 │              │ deviceId=my-node-device            │                  │
│                                      │                  │              │ deviceNumId=2812664419074966       │                  │
│                                      │                  │              │ deviceRegistryId=my-registry       │                  │
│                                      │                  │              │ deviceRegistryLocation=us-central1 │                  │
│                                      │                  │              │ projectId=glass-potion-314513      │                  │
│                                      │                  │              │ subFolder=                         │                  │
└──────────────────────────────────────┴──────────────────┴──────────────┴────────────────────────────────────┴──────────────────┘

So everything went well.

However, where is this data actually stored and how can I access for use in downstream processes, such as analytics, etc? I've explored the Cloud IOT UI but can't find anything there.

Christopher Mills
  • 711
  • 10
  • 28

1 Answers1

3

Cloud IOT does not dictate you the way your devices data is stored.

Please, consider review the IoT technical overview in the Google Cloud Architecture center and some of the proposed architectures, especially Designing a Connected Vehicle Platform on IoT Core.

The IoT Core main documentation provides a very handy diagram as well:

enter image description here

All these architectures are generally based on the processing of the information received from your devices in Pub/Sub in some or other way.

Pub/Sub is a message broker, similar - but with subtle differences - to other systems like Kafka, Active MQ, etcetera. It allows you to decouple and at the same interconnect different services in the Google Cloud Platform (different services can publish information to a certain topic, while other can consume it). But it will not provide a way for message persistent storage on its own: by default, the message retention period is at most 7 seven days.

The responsibility for storing those messages if required lies with the services consuming those messages.

You can use any service that can consume messages from those Pub/Sub topics, but typically you will use products like Dataflow, an advanced stream or batch data processing tool, that will allow you to process your device data and to distribute it to different storage services if necessary.

If your are interested in analytics, the way to go should be either Bigtable or better BigQuery. There are several Dataflow templates available out of the box for storing your Pub/Sub information in BigQuery. Please, consider review this excellent related article as well.

You can store your device information in other services as well, like Cloud SQL, managed relational database services versions of MySQL, PostgreSQL and SQL Server, Cloud Firestore (formerly Datastore), a flavor or NoSQL database, or even Cloud Storage, for storing your device information as plain, unstructured objects. Although a bit dated now, please, see the section What Storage type? in the extraordinary grumpygrace.dev flowcharts page for more information about the different storage options available.

jccampanero
  • 50,989
  • 3
  • 20
  • 49
  • Thank you for your comprehensive answer. I've gone through your links, which has brought a lot into focus. At the root of my miscomprehension was not knowing that Pub-Sub was a standalone service. Hence me searching around in the Cloud IOT UI and docs for more info. – Christopher Mills May 23 '21 at 20:09
  • 1
    You are welcome Christopher. I am happy to hear that the information provided in the answer and the links was helpful. I understand: I think that in this case they are two products very closely related and that usually appear together in the docs and in the actual architectures and use cases. Thank you to @guillaumeblaquiere for his comment, it makes me realize that probably your problem was related with a miscomprehension of Pub/Sub. Please, do not hesitate to contact me if you think that I can be of help. – jccampanero May 23 '21 at 21:21