0

I have 2 simple pipelines in Azure Iot Edge. Now, I am interested in accessing the timestamps added to an iot message at each step/node in the pipeline. For example the 'enqueuingTime' at IoTHub and ASA and 'processedTime' from IoTHub and ASA etc. The following are the pipelines:

  1. IoT Edge Device (input data source) -> IoTHub -> Custom Endpoint to Azure Storage + Route -> Azure Blob

    So far, I have found that each iot message is appended to a blob file with a property EnqueuedTimeUtc and SystemProperties.enqueuedTime, in the header. Also, both displays same value. Is this the time when the message is received in the IoTHub or the time when it is internally processed in the IoTHub?

    Moreover, if I use a custom endpoint into Azure storage, messages are batched into a single blob file. Is it possible to access the timestamp when a single message leaves the IotHub and also when it is written to the blob file? (i.e. I want to get something like a IoTHub-Processed-Timestamp or a Blob-Insertion-Timestamp of individual messages.)

  2. IoT Edge Device (input data source) -> IoTHub -> Azure Stream Analytics -> Azure SQL Database

    In this case, when I am defining the ASA query, the input is the IoTHub endpoint Messaging. In the Select statement, I can access the whole message header and so, I can also access EventEnqueuedUtcTime and EventProcessedUtcTime, which are the timestamps of message enqueue and process completion inside the Stream Analytics job. Further I can access IoTHub.EnqueuedTime, which I assume gives the timestamp when the message is enqueued in the IoTHub. Is there a way to obtain the timestamp when a message enters from ASA and gets inserted into SQL DB? So far, I am using GETDATE() to automatically attach a time stamp during insertion of a record. Is this a good idea?

    Can anybody please let me know if I have a correct understanding of Azure Iot timestamps? Are there any documentations on how to access all such timestamps?

Anirban Das
  • 113
  • 2
  • 6

1 Answers1

2

EventEnqueuedUtcTime is the default timestamp of events coming from an IoT Hub in Stream Analytics is the timestamp that the event arrived in the IoT Hub. EventProcessedUtcTime is the date and time that the event was processed by Stream Analytics. More info this document. As you mentioned when you use stream data from an IoT Hub, you have access to the EventProcessedUtcTime and EventEnqueuedUtcTime in your Stream Analytics query.

When a single data stream contains multiple event types having timestamps in different fields, you can now use TIMESTAMP BY with expressions to specify different timestamp fields for each case.ASA is a temporal system, so every event that flows through it has a timestamp. A timestamp is assigned automatically based on the event's arrival time to the input source but you can also access a timestamp in your event payload explicitly using TIMESTAMP BY.

Update: In addition, EventEnqueuedUtcTime is the date and time that the event was received by the IoT Hub. But IoTHub.EnqueuedTime is the time when the message was received by the IoT Hub.IoTHub.EnqueuedTime is not supported on the telemetry path when a Device/DeviceClient sends a telemetry message to IoTHub.Azure IoT Hub provides the capability to stream data from your connected devices, and integrate that data into your business applications. IoT Hub offers message routing and event routing for integrating IoT events into other Azure services or business applications.More info here.

Michael Xu
  • 4,382
  • 1
  • 8
  • 16
  • So it seems the difference between `EventEnqueuedUtcTime` and `EventProcessedUtcTime` is the total time from the message entering IoTHub and exiting ASA. It appears that with something like `SELECT EVENTPROCESSEDUTCTIME as asa_processedtime, EVENTENQUEUEDUTCTIME as hub_enquedutctime, System.Timestamp as asa_enquedtime INTO Output FROM input ` , the `hub_enquedutctime` and `asa_enquedtime` columns will have same timestamps. In defining ASA query, how can I then access the exact timestamp when the event enters ASA system, using `TIMESTAMP BY`? – Anirban Das Jun 13 '18 at 17:19
  • Also, @Michael can you please explain what is the difference between `EventEnqueuedUtcTime` and `IoTHub.EnqueuedTime` as they seem to have different values. [Documentation](https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-define-inputs#stream-data-from-iot-hub) says the first one is when _event_ was received by the IoT Hub and later is when _message_ is received by IoTHub. I am confused about the difference. – Anirban Das Jun 13 '18 at 17:44
  • 1
    @AnirbanDas, I have updated my response. Generally, message and event are different in IoT Hub. – Michael Xu Jun 14 '18 at 05:27
  • From timestamps, I see that messages and events are different in the system, but I could not find any documentations on what is the difference. This is what I want to know. Sorry if my question was ambiguous. Also, can you please say something about the first comment? – Anirban Das Jun 14 '18 at 14:32
  • 1
    @AnirbanDas, there is no related document to detail the difference between message and event. But you can search 'Azure Event Hubs vs Azure Messaging' to get some information. For the first comment, TIMESTAMP BY is used as a custom timestamp to ingest events out of order with respect to their timestamps. You can see this [article](https://msdn.microsoft.com/en-us/azure/stream-analytics/reference/timestamp-by-azure-stream-analytics?f=255&MSPPError=-2147217396) to understand. – Michael Xu Jun 15 '18 at 05:31