0

We need to get data from 1000s of IOT devices (temperature, pressure, RPM etc total 50+ parameters) and show it on a dashboard without much processing (just checking if numbers are in range otherwise raise alarm) but real time. I have reviewed and tested many aws blog resources like Kinesis Storm ClickStream App

however I think using storm is an overkill for such an easy task. All I want to do is save the data in DB and show graphs (30 Minute, 1 Hour, or custom date). This is what I have figured so far

Device -> AWS IOT(mqtt) -> Kinesis -> x -> dynamoDB -> Presenter Web APP (Laravel) I might have to use Node.js and Redis Pub/Sub as mentioned in ClickStream example for real time updates to graphs and alerts.

I don't want to use Apache Storm because it's in Java and have learning curve (and couldn't find any good resource). I know I can use Lambda but not sure how will it scale.

  • any thoughts on solution ?
  • AWS don't have KCL for PHP, alternatives or solutions? because I am familiar with PHP but not with Java.
Junaid
  • 1,004
  • 8
  • 24

2 Answers2

1

Apache storm is a distributed event processing framework. In your use-case, you do not seem to perform any computation on the events. Basically, your application is doing three tasks:

  1. Ingest data into the system.
  2. Read the data from period X to Y.
  3. Draw graphs on a web frontend.

The ingestion part is taken care by AWS-IOT. The first step you should do is create an SNS topic and publish all IoT data to SNS topics. Here you get the flexibility to create one topic per datatype(ex: temperature, pressure) and attach consumer SQS queues to the topics to accumulate messages into. For a persistent DB, one consumer can be DynamoDB table, another consumer can be a Lambda function which performs some kind of filtering and data transform and updates your cache. If you need to perform some kind of OLAP/Analytical queries on the data, then consider using Redshift as one of the consumers. You will have to get into specific requirements to finalize your design.

Adi
  • 4,149
  • 4
  • 25
  • 41
  • thanks @Adi, so what you are saying is that I remove the kinesis altogether and use 'Device(s) -> AWS IOT(mqtt) -> SNS -> SQS -> Lambda -> dynamoDB'. you also said "one consumer can be DynamoDB" what does that mean? is dynamo capable of saving data to some table directly or will i need lambda ? – Junaid Jun 03 '16 at 07:38
  • Kinesis and SQS are just two different queuing solutions which have slightly different usecases. Kinesis is best suited where you need durability of data and ingestion is in form of a data stream. I wont go into the SQS vs Kinesis discussion and you can evaluate it yourself. "one consumer can be DynamoDB": When you publish messages you any SNS topic, you can have any number of consumers to listen on that topic. Therefore, it behaves as 1...N message delivery. Read more about SNS. – Adi Jun 04 '16 at 22:01
0

Have you considered routing your data to AWS IoT Analytics after receiving the mqtt message in IoT Core? This way you could get rid of all the infrastructure heavy lifting with kinesis, Dynamo and your presentation layer.

AWS IoT Analytics provides you the ingestion, data preparation and querying capabilities. Once you have the data stored in the processed datastore, you can visualize it with AWS QuickSight.

Jurgen
  • 1,243
  • 8
  • 9
  • yes, that was my first option too. but the hardware couldn't support the encryption requirements. – Junaid Jul 01 '18 at 10:57
  • Would you mind elaborating a little bit on this? Instead of pushing the data from AWS IoT to Kinesis ("Device -> AWS IOT(mqtt) -> Kinesis"), you should be able to simply select IoT Analytics as target like this: "Device -> AWS IOT(mqtt) -> IoT Analytics -> AWS QuickSight". See https://docs.aws.amazon.com/iot/latest/developerguide/iotanalytics-rule.html for more details. – Jurgen Jul 02 '18 at 17:11
  • sorry I meant I didn't go this path because devices couldn't send data to AWS IOT in first place (encryption requirement on IOT) so I had to do Device -> mqtt server -> kinesis. – Junaid Jul 11 '18 at 17:27