2

As I understand reading through mesos documentation, is a resource offer is done to a application/framework and it is upto the application to accept/reject offer.

I have a "never-ending" spark streaming app where I configured the executors/cores/memory I need parallelism. Aren't these resources acquired only once when my spark-app starts-up. That is, lets say, if my executors are idle are they handed back to mesos?

Does resource offer and acceptance happens only once in case of spark-streaming?

The same question can be extended for other long-running framework such as cassandra or YARN on mesos.

my understanding is that when spark-streaming is run on coarse-grained model resource exchanges happens once and resources are dedicated to executors for lifetime of spark app

brain storm
  • 30,124
  • 69
  • 225
  • 393

1 Answers1

0

The best source for Spark on Mesos would be the spark docs site here. In the coarse-grained section you can see the following which answers your question:

The benefit of coarse-grained mode is much lower startup overhead, but at the cost of reserving Mesos resources for the complete duration of the application. To configure your job to dynamically adjust to its resource requirements, look into Dynamic Allocation.

If you look into Dynamic resource allocation you can potentially move around executor resources via a Spark Shuffle service. This can be achieved via the provided script by the Spark service, or via Marathon.

ashahan
  • 166
  • 3