2

In Google Cloud, I am having an IoT Core Registry with a number of devices (connected over MQTT) and 2 Pub/Sub topics: events topic (for device telemetry events) and state topic (for device state events). I am processing messages from the events topic with a Dataflow job.

Question: how do I know which device the given message is coming from? Do I need to manually enrich the message (on the device side) with some kind of device ID, or is it somehow provided by the Google Cloud?

What if I used Cloud Functions instead?

In the end I would like to be able to store the messages in a data store (BigTable, BigQuery...) along with its source device ID.

Thanks!

stf
  • 591
  • 1
  • 4
  • 19
  • I imagine you need to pass a DeviceID. Only other information Google could have potentially is IP address which if dynamically linked may not be a guaranteed 1:1 match. – Chase R Lewis May 02 '19 at 15:46

3 Answers3

3

I forget if we attach the device ID as meta data to the Pub/Sub message. You might try to dump the properties of the Pub/Sub object passed to Dataflow to check?

The EASY way to do it is for sure to simply pass the deviceID as an added value in the telemetry blob, and is how I do it (mostly because I don't want to mess with other APIs) and I don't have network size concerns for most of what I do. That would be the consideration. If you're trying to keep your network traffic to the absolute bare minimum. If that's not a hard requirement, I'd just pass it along as extra data in the send blob.

Gabe Weiss
  • 3,134
  • 1
  • 12
  • 15
1

If u are planning to use Cloud Function, you can find the device name/ID from attribute['deviceId'] information in the event['attributes']. Attached a Python Cloud Function triggered by PubSub example,

def cloud_function(event, context):
    attribute = event['attributes']
    attribute['deviceRegistryLocation']
    attribute['deviceRegistryId']
    attribute['deviceId']
Sian En Ooi
  • 61
  • 1
  • 5
0

You probably want to connect each device to a different topic, just remember each registry can have max 10 topics, you can have subfolders to each topic too.

I personally like to use cloud functions to parse and or pass telemetry data to somewhere else, for example: BigQuery or Firebase Real time Database....

Juan Quintero
  • 53
  • 1
  • 10
  • Are you referring to MQTT or Pub/Sub topics? What is the limit on subfolders? – stf May 03 '19 at 09:44
  • Pub/Sub Topics; make a test... you can have: device1.telemetry/subfolder1 (this is a Pub/Sub topic with a subfolder) I personally used this to perform some tests but. In the cloud functions Im not sure how to handle it, cant remember but there is for sure a way.... – Juan Quintero May 06 '19 at 13:56
  • Connecting each device to a different topic will not scale. Cloud Pub/Sub only allows 10,000 topics per project. The subfolder mechanism is meant to group different kinds of data from all devices into different topics. – Kamal Aboul-Hosn May 07 '19 at 23:24