I have a lot of telemetry data being sent by various sensors and it often happens that multiple parameters for a single device is changed within the same second (with millisecond differences). I want a method to merge these records for the same second in Azure stream analytics
I tried to add a trigger to the main SQL table,
ALTER TRIGGER [dbo].[trgOEESignals_updated_upsert] ON [dbo].[TBLOEESIGNALS] INSTEAD OF INSERT
AS
BEGIN
set nocount on;
MERGE TBLOEESIGNALS AS old
USING inserted AS new
ON LTRIM(RTRIM(new.SITE_CODE)) = LTRIM(RTRIM(old.SITE_CODE)) and
LTRIM(RTRIM(new.MACHINENAME)) = LTRIM(RTRIM(old.MACHINENAME))
and DATEDIFF(SECOND, old.SIGNALRECEIVEDTIME,new.SIGNALRECEIVEDTIME) = 0
WHEN MATCHED THEN
UPDATE SET
old.REC_READ = 0,
old.CURRENTMODE = new.CURRENTMODE,
old.CURRENTSPEED= new.CURRENTSPEED, -- same logic for here
old.TOTALPARTS = new.TOTALPARTS,
old.PRODUCEDPARTSPERCYCLE= new.PRODUCEDPARTSPERCYCLE,
--- add a condition for cycle counter if 1 is present in existing or in the new one
old.CYCLECOUNTER= new.CYCLECOUNTER
WHEN NOT MATCHED THEN
INSERT(signalreceivedtime,currentmode,currentspeed,totalparts,machinename,
currentjob,producedpartspercycle,rec_read,site_code,scrap,
cyclecounter,scrap_code,target_cycle_time)
VALUES (new.SIGNALRECEIVEDTIME,new.CURRENTMODE,new.CURRENTSPEED,new.TotalParts,new.MachineName, COALESCE(new.CurrentJob, ''), new.PRODUCEDPARTSPERCYCLE,
new.rec_read, new.site_code, new.scrap, new.cyclecounter, new.scrap_code, new.target_cycle_time);
END;
but still the same issue persists. I have noticed, there is a ASA function output also doing the similar approach and first wanted to try it directly on SQL to see the results. The sensor readings are continuous stream by nature and I dont know a good approach for now