1

I have a question for you. I am new in AWS IoT, AWS ES and MQTT. I followed this nice tutorial and finished it. I sent a JSON data to AWS IoT by using AWS Lambda function. Then, in AWS IoT I created a rule to send this data to AWS Elasticsearch. In the end, I visualized the data by using Kibana.

My next task is storing this data in 3 different types of databases: Relational DB (Amazon Aurora), Key Value DB (Amazon Dynamo DB) and Document DB (Amazon Document DB). But as you know, AWS IoT doesn't provide these options as a rule. It only provides "Insert A Message Into DynamoDB Table" option. In that case, how can I create other databases and send data from AWS IoT? Is there any source or tutorial that you can suggest for me about that?

I would be very happy if experienced people can guide me Thanks a lot!

(FYI: My elasticsearch doesn't have a VPC. I am using public access. So far, I didn't use any EC2.)

Mark B
  • 183,023
  • 24
  • 297
  • 295
firefighter
  • 171
  • 1
  • 14
  • You'll need to do some programming within your Lambda to put data into the three data stores. What programming environment would you normally use? – stdunbar Oct 26 '20 at 18:44
  • @stdunbar I can use Python, Java, Javascript. Do you have any example Lambda function like this? – firefighter Oct 27 '20 at 10:51

2 Answers2

3

The most common method of inserting AWS IoT data into a database is to configure IoT to send the messages to an AWS Lambda function. Inside the Lambda function you would connect to your database and insert the data just like any other database interaction.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • Are you sure? for high rate of data - you would hit the invocations limit - it might be better to batch them into Firehose -> s3, and then process them in batches. Also, if the IOT lambda fails, you are going to lose your message. – ArielB Mar 07 '21 at 08:06
0

You can use AWS IoT Analytics to store messages in an S3 bucket. A simple pipeline (with no transformations) will store incoming messages in a compressed format, and the output in json.gz files named by data/timestamp. If you configure you IoT Analytics with custom S3 Buckets, you can set the retention policy of the buckets to expire old data.

The you can write some code to read the Analytics Pipeline output files and put them into whatever data store you want.

Or you can just send the messages to a Lambda function and have them connect and store messages in a database. The advantage of using an Analytics Pipeline is you can continue to recieve/store MQTT messages if you've got database downtime, or you deploy database ingestion code with bugs.

Wheat
  • 845
  • 4
  • 12
  • Please can you elaborate on "The advantage of using an Analytics Pipeline is you can continue to recieve/store MQTT messages if you've got database downtime, or you deploy database ingestion code with bugs." - Do you mean that the pipeline will queue all messages sent to the lambda if the database is down? – variable Nov 03 '20 at 09:38
  • The pipeline will continue to store IoT messages in it's S3 Bucket. You would need to implement your own logic in your Lambda to read messages from that S3 Bucket and ingest them into a database, etc. For example, you could create an SQS queue that gets populated with S3 Objects stored in the pipeline output S3 Bucket, then your Lambda can pop items off the queue after they are successfully ingested. – Wheat Nov 03 '20 at 16:13