I am facing some delay in event processing through stream analytics. I am working on an IoT Project and gateway is sending messages to IoTHub. Since we are dealing with different type of messages, through event hub end point we are routing the messages. Our Stream analytics job will pick the events from event hub and will start processing and pushing to Service Bus Queue..
After starting the job for the next minute on wards messages are pushing to service bus queue. So we compared the time difference between event hub utc time and stream analytics process time. Initially it will be executing within 5 to 10 seconds.
But after 1 hour this delay will be getting incresaed to 15 seconds, and it will continue. So after 6 or 7 hours, the message produced and message deliver to Service Bus queue gap will be more than 1 minute. Since our application is directly based on realtime data, I may need to restart the job after 7 hours(Which is not a permanent solution).
In the event hub I am using 4 partition and 3 streaming units are using. Since streaming unit utilization is very low(16%), so increasing streaming unit is not a solution for this problem.
Kindly help on this issue.
My Stream Analytics Query Giving Below:
WITH
rawmessage AS
(
SELECT
digitaleventhubstreaminputonlineclassaforrawdata.*,
GetMetadataPropertyValue(digitaleventhubstreaminputonlineclassaforrawdata,
'EventHub.IoTConnectionDeviceId') as iotdevice
FROM digitaleventhubstreaminputonlineclassaforrawdata
Partition By PartitionId
)
,
messagetoprocess AS
(
SELECT
rawmessage.*,
digitalblobreferenceinputnmea.*,
digitalblobreferenceinputwidget.*,
digitalblobreferenceinputscalingfactor.*
FROM rawmessage Partition By PartitionId
LEFT JOIN digitalblobreferenceinputnmea ON rawmessage.vessel_id=digitalblobreferenceinputnmea.vessel_id
LEFT JOIN digitalblobreferenceinputwidget ON rawmessage.vessel_id=digitalblobreferenceinputwidget.vessel_id
LEFT JOIN digitalblobreferenceinputscalingfactor ON rawmessage.vessel_id=digitalblobreferenceinputscalingfactor.vessel_id
WHERE rawmessage.sensorval IS NOT NULL
)
,
processedmessage AS
(
SELECT
event.vessel_id as vessel_id_fk,
event.iotdevice as device_id_fk,
event.PartitionId as partitionId, UDF.getEpochTime(event.EventProcessedUtcTime)as EventProcessedUtcTime,
UDF.getAnalyticsProcessTime('arg') as AnalyticsProcessTime
FROM messagetoprocess as event Partition By PartitionId
)
--output is writing into service bus queue for socket push
SELECT * INTO digitalqueueoutputonlineclassasocketdata FROM processedmessage Partition By PartitionId
