0

I log simple tabular data: ISO8601 date/time, number, number, text
Once a day a small lambda job downloads it as a CSV and runs a few simple stats on it and emails me. Once every few days, I might need to visually glance at something that might have happened earlier that day.

Currently, I'm pushing the events as they happen to an SQS pipeline, then popping them off the queue in a batch lambda job once every 10 mins to a Google sheet. It's easy to just click a bookmark from any browser: there's the data, nice and easy and tabular, and it provides native csv export.

But I just hit the Google Sheets 500,000 cell limit, and now that everything is soon to run from one EC2 server, I might as well go all-AWS. And the thing which served me well as a sticking plaster for 3 years is now feeling horribly clunky.

I looked at Cloudwatch, but it appears that it can't export to CSV without fiddling via jq (?) I looked at https://aws.amazon.com/products/databases/ - dynamob DB is out as you can't glance at the output in chronological order. Perhaps simpleDB is the solution? https://aws.amazon.com/simpledb/

But again, I don't need a DB. Just something that can cheaply and simply receive about 50 lines an hour, in an sqs manner, has no row limit, allows easy automated export to CSV via the AWS node sdk, and will let me cast a human eye over a few rows from time to time like a spreadsheet.

I keep coming back to Cloudwatch because of CloudWatch Logs Insights. It's only the lack of csv export with the sdk that's putting me off.

Am I looking in the right direction? Have I missed something obvious? Thanks.

digitaltoast
  • 659
  • 7
  • 23
  • 1
    maybe GDS can be of any use as a replacement for GS which has those unpleasant cell limitations - https://datastudio.google.com/u/0/navigation/reporting – player0 Dec 20 '20 at 19:47
  • 1
    Thank you - I have to confess that I had no idea of the existence of GDS, even though it appears to have launched 2 years ago! Thank you - seems like the API allows a write in the format I want, I think I might even be able to do away with my nightly node processing script if the connectors work like they seem to. Thanks again! – digitaltoast Dec 21 '20 at 14:08

0 Answers0