0

Is it recommended to separate the projects that's used for ETL purpose ( Aggregating > 1TB data into a Table) and ( Querying data < 1TB ) Visualization Purpose.

Or will the BI Engine smartly does this? How long will BI Engine cache queried data if it does at all? e.g.

  1. None at all it will just use the reservation for in memory processing
  2. It will cache the data until it becomes full and discard the oldest data
Anthony Liu
  • 343
  • 1
  • 12

1 Answers1

0

It is recommended to separate projects the way you described. You then create BI Engine reservation for the visualization project to cache the data.

For your visualization project, BI Engine will store the most recent data in RAM and offload old data over time. There are no specific guarantees on time the data leaves in cache today - BI Engine will adapt to your workload.

Couple of things that may change in future:

  • Ability to choose which tables can use BI Engine reservation. That way you can store data in the same project with ETL, but pick tables that should be cached in BI Engine.
  • We may add some guarantee on how long the data is cached.
Alexey Klishin
  • 174
  • 8