0

I have a pretty complex pipeline that does several machine learning algorithms on a given data. One of the algos uses GPU for processing. I'd like to somehow distinguish several nodes of my cluster with available GPU units so the algo would run only on those partitions. Is it possible with Spark?

I understand that it's more or less uncommon to have specific resources in a general computation environment, but this algo runs once in a while and it stays still whole other time - I'd like to use these nodes, too.

evgenii
  • 1,190
  • 1
  • 8
  • 21
  • Which resource manger do you use? – zero323 Dec 04 '15 at 10:10
  • @zero323 I was thinking about standalone right now, but we have a potential client and we will use either YARN or Mesos. If one of them is capable to do that we'll take it. – evgenii Dec 04 '15 at 15:52
  • As far as I know not directly. And definitely not by transformation basis.You should be able to use [Messos attributes](https://mesos.apache.org/documentation/attributes-resources/) and handle the rest at the pipeline level (Oozie, Luigi) though. – zero323 Dec 04 '15 at 16:02
  • @zero323 thanks a lot, I'll take a look. – evgenii Dec 04 '15 at 16:04

0 Answers0