4

I am working on a project where I need to share execution state across different spark application.

I decided to go with apache-ignite as a shared memory storage between different spark application.

I was thinking of going with embedded ignite mode with static allocation in spark where ignite nodes will start in Spark executor process. So that, tasks will be executed in the same process where Data is present. But, this mode is deprecated.

I could go with standalone Ignite deployment but there would be inter-process communication to get and save the state which I want to avoid.

Is there any way to tell the Spark to create its executors in already present process (in this case, Ignite nodesprocesses) ?

Can ExternalClusterManager be implemented to achieve this ?

Does Ignite is planning to introduce such mode in future ?

Amar Gajbhiye
  • 484
  • 5
  • 17

1 Answers1

3

Well, yes, your general direction is reasonable. Ignite's deprecated embedded deployment is, so to say, embedded "backwards" - when you embed Ignite into Spark it works poorly, but if we embedded Spark into Ignite, it would work better.

Yes, I assume it would be possible to implement. It probably could even be implemented outside of Ignite.

I don't think there are any open issues for that in Ignite backlog, but you can share you suggestions on Ignite dev mailing list.

And now the main part. All you're going to achieve with your suggestion is replacing inter-process communication with intra-process. Usually, communication on the same host isn't that expensive. You might see some performance gain from this but I'd only went into implementing this if there were a solid evidence that this is going to solve a real problem.

Stanislav Lukyanov
  • 2,147
  • 10
  • 20