-1

A system is to be designed where a number of sensors will transmit data to event hub. Each sensor will transmit number of data(max 15-20 (around 5 KB)) each second.

  • One use case is that all of these raw data has to be transmitted to cosmosDB with minimum/no latency which i could achieve with below architecture.

Raw Data -> Event Hub -> Azure Stream Analytics -> CosmosDB

Note: The above use case is to determine connection status of sensor. So it has to be as quick as possible.

  • Another use case is the indication property of sensor which is dependent on the last values from the sensors (at max previous 200 values). Now the problem is here. I tried using the following architecture.

Raw Data -> Event Hub -> Azure Stream Analytics -> Azure Function -> CosmosDB

So here, output from stream analytics goes to azure function where it gets the previous 200 values from cosmosDB does necessary calculation and then store the result again back to cosmosDB. But this process seems to be very slow.

COSMOS DB CONTAINERS
//Sensor Property Container
{ 
  id: "sensor_id",
  connection:true, //This needs to be updated as soon as raw data is available
  indication:"RED" //This depends on previous 200 values from sensor
}

//Sensor Raw Value Container
{ 
  id: "sensor_id",
  rawData: "RAW_VALUE",
}

You can look at image below of what i am actually looking for.

Data Flow Image

I also tried adding user defined function in azure stream analytics to run algorithm within but it doesn't support cosmosDB as reference data.

Nabin Shahukhal
  • 1
  • 1

1 Answers1

-1

You should try to incorporate the logic in Stream Analytics. Adding more moving parts always leads to latencies. Can you do a count of values received every minute grouped by deviceId? and generate alert when this count less than expected?

Krishna Mamidipaka
  • 7
  • 1
  • I am grouping the values by device Id for a second and only getting the latest value. I have tried to incorporate the logic in Stream analytics. However, every second when a new data arrives, i need the previous status which was saved in cosmosdb. So i will need to read data off the cosmosdb every time new data arrives. I am not sure how this can be done in stream analytics. – Nabin Shahukhal Dec 12 '19 at 22:47