0

I only know the basics of Apache Spark framework. Is it possible to extend Spark scheduler to place its task's operators on different node machines like it is possible in Apache Storm? On Apache Storm is only implement the IScheduler and its methods.

Felipe
  • 7,013
  • 8
  • 44
  • 102
  • 1
    If the question is if the API is intended to be extended by the end users, then the answer is most likely negative. And to [quote](http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-SPARK-24615-SPIP-Accelerator-aware-Scheduling-tp26583p26606.html) Felix Cheung response to acceleration aware scheduling - _Third, the proposal touches on some fairly core and sensitive components, like the scheduler, and I think more discussions are necessary. We have a few comments there and in the JIRA._. Have you rather looked at prefferedLocations? – user10938362 Mar 23 '19 at 10:22
  • Hi @user10958683, thanks for your answer. Seems Flink also have this requirement (https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit#heading=h.k15nfgsa5bnk). About "prefferedLocations", I don't know what it is in Spark. Where I can find some reference? I searched and couldn't find. thanks – Felipe Mar 23 '19 at 10:47
  • 1
    Take a look at [this answer](https://stackoverflow.com/a/47801511/10958683) by [Jacek Laskowski](https://stackoverflow.com/users/1305344/jacek-laskowski). – user10938362 Mar 23 '19 at 11:44

0 Answers0