0

I loaded same amount of data when native persistence enabled and disabled. I noticed that when native persistence is enabled off-heap memory usage increased x4 times. Is it normal?

Ayberk Cansever
  • 3
  • 2

1 Answers1

0

When you have native persistence there's potentially more overhead and also there's checkpointing page buffer. However, it is fully up to you to determine how much RAM will be dedicated to off-heap storage. Just tune your DataRegion settings.

alamar
  • 18,729
  • 4
  • 64
  • 97
  • i must have all data in memory and i must have all data persistent. very interestingly 1GB off-heap turned into 4GB off heap after enabling native persistence. It is, 300%, very very abnormal overhead i think. – Ayberk Cansever Dec 14 '18 at 12:16
  • @AyberkCansever the whole point of persistence is to avoid having all data in memory. 4GB is a small-ish data-region as far as Apache Ignite is concerned, I can see how it might be dominated by metadata, etc. Maybe you can get this number down by changing approach, maybe not. – alamar Dec 14 '18 at 12:22
  • I want very fast queries and also i don't want to lose my data after restarts. My total data will be 500GB in fact. After enabling native persistence i will need 2TBs of RAM. On the other hand, with 500GB, i will lose speed because my all data will not fit into memory. What kind of extra overhead comes with native persistence? For ex. decreasing object count will affect memory consumption? – Ayberk Cansever Dec 14 '18 at 12:27
  • If you have 500GB of data the overhead will not be 4x as you observe with 1G of data. It would be much smaller. For example, when persistence is enabled, Durable Memory cannot be shared between different caches. Maybe there are more examples of potential overhead. – alamar Dec 14 '18 at 12:54
  • Ok understood. i guess it is not possible to predict the total memory usage without going to production :) linear calculation seems to be wrong. – Ayberk Cansever Dec 14 '18 at 13:20