1

I am trying to use AWS Timestream to store data with timesteamp (in python using boto3).

The data I need to store corresponds to prices over time of different tokens. Each record has 3 field: token_address, timestamp, price. I have around 100 M records (with timestamps from 2019 to now).

I have all the data in a CSV and I would like to populate the DB with it. But I don't find a way to do this in the documentation as I am limited by 100 writes per query according to quotas. The only optimization proposed in documentation is Writing batches of records with common attributes but in my my case they don't share the same values (they all have the same structure but not the same values so I can not define a common_attributes as they do in the example).

So is there a way to populate a Timestream DB without writing records by batch of 100 ?

Vince M
  • 890
  • 1
  • 9
  • 21
  • I never tried this in Python, but other SDKs support writing data using the https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/timestreamwrite/TimestreamWriteClient.html – smac2020 Jan 31 '23 at 14:43
  • Well, If I understood your question correctly, the `token_address` column should be "group-able". You could split the writes of price/time by token_address. I cannot see another way of inserting that many data other than batching. A caveat, though: It should be fairly fast. Have you tried it? – Eduardo Elias Saléh Feb 07 '23 at 10:53

1 Answers1

0

I asked AWS support, here is the answer:

Unfortunately, "Records per WriteRecords API request" is a non-configurable limit. This limitation is already noted by the development team.

However, to get any additional insights to help with your load, I have reached out to my internal team. I will get back to you as soon as I have an update from the team.

EDIT:

I had a new answer from AWS support:

Team, suggested that a new feature called batch load is being released tentatively at the end of February (2023). This feature will allow the customer to ingest data from CSV files directly into Timestream in bulk.

Vince M
  • 890
  • 1
  • 9
  • 21