I'm using the Javascript AWS S3 SDK to extract data from a CSV on my server. This is done via below SQL query statement:
SELECT timestamps, parameterX
FROM S3Object
WHERE ${timestamp_header} > '${startTime}' and ${timestamp_header} < '${endTime}'
This returns correctly the timestamps
and parameterX
values for the period between startTime
and endTime
.
However, I would like to be able to dynamically extract these values from the last X seconds of the dataset (e.g. named windowSec
). In other words, if the last (aka maximum) timestamps
value in the CSV is 2019-04-17 19:50:55
and I specify I want the last 20 seconds, I'd want observations between 2019-04-17 19:50:35
and 2019-04-17 19:50:55
.
I'm able to do this by running an initial S3 select statement to get the full timestamps
column and extracting the maximum value of this. But I assume there's a "proper" way of doing this in the S3 select query directly. I've tried below, but I'm unable to get it to work:
SELECT timestamps, parameterX
FROM S3Object
WHERE ${timestamp_header} > (MAX(${timestamp_header})-windowSec)
and ${timestamp_header} < MAX(${timestamp_header})
Any suggestions would be appreciated.
CSV sample data (full dataset):
timestamps,parameterX
2019-04-17 19:49:15.673949957+02:00,4.726368587
2019-04-17 19:49:20.673949957+02:00,4.71613058
2019-04-17 19:49:25.673949957+02:00,4.705892573
2019-04-17 19:49:30.673949957+02:00,4.695654566
2019-04-17 19:49:35.673949957+02:00,4.685416559
2019-04-17 19:49:40.673949957+02:00,4.675178552
2019-04-17 19:49:45.673949957+02:00,4.664940546
2019-04-17 19:49:50.673949957+02:00,4.654702539
2019-04-17 19:49:55.673949957+02:00,4.644464532
2019-04-17 19:50:00.673949957+02:00,4.634226525
2019-04-17 19:50:05.673949957+02:00,4.623988518
2019-04-17 19:50:10.673949957+02:00,4.613750511
2019-04-17 19:50:15.673949957+02:00,4.603512505
2019-04-17 19:50:20.673949957+02:00,4.593274498
2019-04-17 19:50:25.673949957+02:00,4.583036491
2019-04-17 19:50:30.673949957+02:00,4.572798484
2019-04-17 19:50:35.673949957+02:00,4.562560477
2019-04-17 19:50:40.673949957+02:00,4.55232247
2019-04-17 19:50:45.673949957+02:00,4.542084464
2019-04-17 19:50:50.673949957+02:00,4.531846457
2019-04-17 19:50:55.673949957+02:00,4.52160845
Intended output when windowSec = 20:
timestamps,parameterX
2019-04-17 19:50:35.673949957+02:00,4.562560477
2019-04-17 19:50:40.673949957+02:00,4.55232247
2019-04-17 19:50:45.673949957+02:00,4.542084464
2019-04-17 19:50:50.673949957+02:00,4.531846457
2019-04-17 19:50:55.673949957+02:00,4.52160845
Edit: Added sample data