I have a script that collects data from somewhere and writes it into a mongodb collection every 10 minutes. Then we have a frontend that displays the historical trends of data in form of graphs/charts etc. I noticed that now we have data for around 2 years. All the data is at a 10 minute resolution. Generally, we like to see data at resolution of 10 minutes for past 6 months only. 6 month to 1 year old data is checked at an hourly resolution while older than an year at a daily resolution.
This means that we should somehow aggregate the 10 minute resolution data to higher resolution if its older than some time. E.g. average out (or may be max/min depending on the parameter) the data older than a year at an hourly basis and remove the 10 entries and make a new single entry.
Are there any frameworks available out there that could support such policy based data management?
