I was attempting to ingest around 100,000 records into a newly created Timestream table through Lambda function. All the records have the same timestamp and the timestamp is beyond the retention period of memory store so they were all written to magnetic store. The records were written to Timestream using boto3 with Python 3.9 in batches of size 100 each. After all 1,000 WriteRecords requests were sent, I noticed that there are only around 20,000 records in the table. Because the writing to magnetic store is asynchronous, no errors were raised during the execution of lambda function. In the error log bucket that receives the error logs of magnetic store, there was no update either.
After going over the AWS Timestream documentation, I suspected that this might be related to the magnetic store partitions. It appears that AWS slowly boots magnetic store partitions when there is a high throughput workload, trying to scale up. Would writing large amount of data to magnetic store cause data loss instead of queueing the incoming data? Is that the expected behaviour?
Background Information:
Lambda function:
- memory allocated: 4096MB
- runtime: Python 3.9
- records processed and ingested using two threads created with threading.Thread
Timestream table:
- memory store retention period: 1 day
- magnetic store retention period: 10 years
Records:
- 100,000 records
- multi-mearsure records with the same timestamp in 2022
- relatively large size for each record
I initally suspected that there might be a throttle rate issue. But when I reduced the speed of sending requests by half, even less records came through. With half speed ingestion, only 8,000 records were present in the table.
I also suspected that there may be records overwriting each other, so I put a unique counter in the dimension for each record before the ingestion. However, the data still lost.
After noticing that the issue might be caused by magnetic store partition. I tried to modify the timestamp of the records and ingested them into memory store, everything worked out fine.
I also tried to ingest the same records twice consecutively into the magnetic store. Although in the first time there were still only around 20,000 records coming through, all 100,000 records were successfully ingested in the second attempt.
