0

as the title suggests, is there any repository where somebody has implemented this kind of constrained optimization method in (py)Spark?

The thing is the project I am working on has got several steps and, while nearly all of them can be done natively in Spark, this one presents a bit of a pickle because, as much as I can code up in (py)Spark the "outer" functions of such method, the "core code", namely the "slsqp" function is apparently written in Fortran which is way beyond my area of expertise (or anybody's really in my company).

p.s.: I cannot install scipy in the cluster as the company I am in right now will most likely not allow it (not at least in a timely matter)

Asher11
  • 1,295
  • 2
  • 15
  • 31
  • I don't about slsqp, but I think you can use scipy by just downloading anaconda in your cluster (as long as you have write permissions in the cluster), then you can use scipy in spark UDFs and do what you want to do – mck Dec 13 '20 at 19:10
  • the problem is just that: I cannot install it any time soon if at all! – Asher11 Dec 13 '20 at 21:08
  • 1
    you dont have to download the scipy in the cluster, instead, ship your own virualenv to the cluster which allows you define whatever dependency to use – E.ZY. Dec 14 '20 at 05:13
  • Yes the fact is I do not manage that directly so I would have to go the bureucratic channels in order to obtain that. Seems to be the only way, so I will just have to be persuasive I guess – Asher11 Dec 15 '20 at 08:55

0 Answers0