0

NReco PivotData Cube processing

This component is new to me, and i am curious whether it supports frequent updates to the data source? Normal size of data will be 1000 rows containing 60 facts and 6 dimensions. Is it possible to update single rows of data? Will this trigger full reprocessing, and how long will it take?

kenander
  • 55
  • 1
  • 4

1 Answers1

0

NReco.PivotData library implements in-memory multidimensional data structure which holds pre-aggregated metrics for dimensions you want to use in your reports. There are 2 different ways how it can be used:

  • ROLAP-style way (mostly used): PivotData class instance is short-living, it is created and populated 'on-the-fly' at the moment when you need to render pivot table or prepare JSON data for the chart. Data source doesn't matter: this can be in-memory list of models that is processed with PivotData.ProcessData method or DB aggregation query results may be loaded with help of GroupedSourceReader class.
  • for real-time reports on append-only dataset, you can load PivotData class instance once on the application start, and then 'update' it with new records. Depending on your needs, PivotData may keep and pre-calculate totals/totals, or 'lazy' roll-ups may be performed only when total value is accessed (this is controlled with PivotData.LazyTotals flag).

Regarding

Normal size of data will be 1000 rows containing 60 facts and 6 dimensions.

this is very small data, 1k rows is processed by ProcessData method very quickly (only dimensions/measures that are needed for the concrete report are calculated). If you use database as a data source, it should be able to execute aggregate (GROUP BY) queries fast enough; specialized analytical DBs can do that in seconds even for billions of rows.

Vitaliy Fedorchenko
  • 8,447
  • 3
  • 37
  • 34
  • Ah. The ProcessData method is also used to append data. Now I see! Tested with 10000 rows, 9 dimensions, but only two facts. ProcessData in 23ms, but also 23ms after appening a single row. – kenander Oct 07 '19 at 13:19
  • Do you know if it is possible to replace a single row by assigning a field of a datarow to be treated as a key? – kenander Oct 07 '19 at 13:43
  • @kenander you can append only new facts to existing PivotData instance. If something is changed in already processed rows, you need to re-process all input facts (not a problem in case of 10k rows). In case of large datasets initial data aggregation should be performed on database level. – Vitaliy Fedorchenko Oct 07 '19 at 13:50