1

I have a large number of Iot Devices that are generating data and sending it to IoT Core through MQTT. I need to store this telemetry data in a timestream table. I am trying to reduce costs as much as I can, while keeping the latency reasonably low. Eventual consistency is enough.

Using the IoT Core Rules Timestream action is out of the question because I need to use Multi-Measure records

Currently, what I'm doing is this : An IoT Core Rule, sends each record to a lambda which then inserts it into the timestream table. This means that the lambda is invoked once per record and data is inserted into timestream, once per record.

I'd like to take advantage of Batching in order to reduce my costs. I'd like to know what are the best practises and patterns for achieving this are. Any help and suggestions are welcome!

Prithivi Maruthachalam
  • 21
  • 3

1 Answers1

0

I’d suggest to either use SQS or Kinesis data stream between iot rule and lambda. Both SQS and Kinesis can work in either batch window or batch size. That way you would reduce cost on the lambda and process more messages per invocation.

I would also highly suggest to use this tool: https://github.com/alexcasalboni/aws-lambda-power-tuning

That is a good way to understand which lambda memory settings is the best for your lambda.

brushtakopo
  • 1,238
  • 1
  • 3
  • 16
  • Thank you! and another thanks for the handy tool. I'd assumed I'd have to use SQS or Kinesis streams. Any suggestions/tips on choosing between the two? – Prithivi Maruthachalam Nov 13 '22 at 13:08
  • It highly depends on the use case. I used to go with SQS all the time before Kinesis supported On-demand. Now, I go with Kinesis on-demand since it support longer data retention. Also, Kinesis supports fan-out, which is interesting if you need to deliver messages faster to consumers. – brushtakopo Nov 14 '22 at 11:04