I am running a query in Stream analytics with a UNION function. I would like to get the distinct values for the query results. Since the UNION is allowing duplicates in Azure Stream Analytics I am getting result with duplicate values. I have tried using DISTINCT keyword also but even it is not working.
Below is the Query I tried.
WITH
ABCINNERQUERY AS (
SELECT
event.ID as ID,
ABCArrayElement.ArrayValue.E as TIME,
ABCArrayElement.ArrayValue.V as ABC
FROM
[YourInputAlias] as event
CROSS APPLY GetArrayElements(event.ABC) AS ABCArrayElement
),
XYZINNERQUERY AS (
SELECT
event.ID as ID,
XYZArrayElement.ArrayValue.E as TIME,
XYZArrayElement.ArrayValue.V as XYZ
FROM
[YourInputAlias] as event
CROSS APPLY GetArrayElements(event.XYZ) AS XYZArrayElement
),
KEYS AS
(
SELECT DISTINCT
ABCINNERQUERY.ID AS ID,
ABCINNERQUERY.TIME as TIME
FROM ABCINNERQUERY
UNION
SELECT DISTINCT
XYZINNERQUERY.ID AS ID,
XYZINNERQUERY.TIME as TIME
FROM XYZINNERQUERY
)
SELECT
KEYS.ID as ID,
KEYS.TIME as TIME
INTO [YourOutputAlias]
FROM KEYS
In the above query ID is unique and Time will be An array of values with Time and value of ABC/XYZ.
input json file is as below.
[
{"ID":"006XXXXX",
"ABC":
[{"E":1557302231320,"V":54.799999237060547}],
"XYZ":
[{"E":1557302191899,"V":31.0},{"E":1557302231320,"V":55}],
{"ID":"007XXXXX",
"ABC":
[{"E":1557302195483,"V":805.375},{"E":1557302219803,"V":0}],
"XYZ":
[{"E":1557302219803,"V":-179.0},{"E":1557302195483,"V":88}]
]
Expected result without duplicates.