-2

I just started learning about IoT and data streaming. Apologies if this question seems too obvious or generic.

I am working on a school project, which involves streaming data from hundreds (maybe thousands) of Iot sensors, storing said data on a database, then retrieving that data for display on a web-based UI.

Things to note are:

  1. fault-tolerance and the ability to accept incomplete data entries
  2. the database has to have the ability to load and query data by stream


I've looked around on Google for some ideas on how to build an architecture that can support these requirements. Here's what I have in mind:

  1. Sensor data is collected by FluentD and converted into a stream
  2. Apache Spark manages a cluster of MongoDB servers
         a. the MongoDB servers are connected to the same storage
         b. Spark will handle fault-tolerance and load balancing between MongoDB servers
  3. BigQuery will be used for handling queries from UI/web application.

My current idea of a IoT streaming architecture : enter image description here
The question now is whether this architecture is feasible, or whether it would work at all. I'm open to any ideas and suggestions.

Thanks in advance!

Ram Ghadiyaram
  • 28,239
  • 13
  • 95
  • 121
Ian
  • 7
  • Sure, try it. Why do you need approval here? You're a student. This is what real learning is. – duffymo May 31 '16 at 12:20
  • Well, try everything. You will learn them all, and you will understand. You are a student after-all. And in SO we ask questions about specific issues rather than getting opinions. You could google those different technologies and stick to one too. – Asim K T May 31 '16 at 12:24

1 Answers1

2

Note that you could stream your device data directly into BigQuery and avoid an intermediate buffering step.

See: https://cloud.google.com/bigquery/streaming-data-into-bigquery

Adam Lydick
  • 1,092
  • 7
  • 15