0

Please help me to understand following

  1. What are the steps to configure OFF-HEAP storage for Apache Spark 2.x ?
  2. Is it possible to configure Alluxio as the OFF-HEAP storage in 2.0 ? Is it removed since 2.x ?
  3. How does OFF-HEAP works with Dynamic Allocation, where executors being added and removed randomly on demand?
dtolnay
  • 9,621
  • 5
  • 41
  • 62
Jais Sebastian
  • 21
  • 4

1 Answers1

0

What are the steps to configure OFF-HEAP storage for Apache Spark 2.x ?

No additional steps are required.

Is it possible to configure Alluxio as the OFF-HEAP storage in 2.0 ? Is it removed since 2.

It is not. External block store has been removed in Spark 2.0: SPARK-12667 - Remove block manager's internal "external block store" API. You can use Alluxio as a normal file system instead.

How does OFF-HEAP works with Dynamic Allocation, where executors being added and removed randomly on demand?

Because OFF-HEAP is internal, it will loose blocks as any other storage level.

Check:

Alper t. Turker
  • 34,230
  • 9
  • 83
  • 115
  • Thanks for the answers. But I have few more doubts here , 1) who manages OFF-HEAP memory ? Is managed by individual Spark Executor ? 2) How does the OFF-HEAP memory allocation reflect to the Resource Managers like Mesos ? Do we need to do any static reservation in Mesos? – Jais Sebastian Feb 13 '18 at 02:29