5

I've got a hadoop cluster that I'm doing data analytics on using Numpy, SciPy, and Pandas. I'd like to be able to submit my hadoop jobs as a zip/tar file using the '--file ' argument to a command. That zip file should have EVERYTHING that my python program needs to execute such that no matter what node my script executes on in the cluster, I won't run into an ImportError at runtime.

Due to company policies, installing these libraries on every node isn't exactly feasible, especially for exploratory/agile development. I do have pip and virtualenv installed to create sandboxes as needed though.

I've looked at zipimport and python packaging but none of that seems to fulfill my needs/I'm having difficulty using these tools.

Has anyone had any luck doing this? I can't seem to find any success stories online.

Thanks!

jbarney
  • 159
  • 1
  • 8
  • Not sure about these particular dependencies but pex works pretty well for creating one file: https://pex.readthedocs.org/en/latest/ – Simeon Visser Jun 14 '15 at 19:18

1 Answers1

0

I have solved similar problem in Apache Spark and Python context by creating a Docker image which has needed python libraries and Spark slave script installed. The image is distributed to other machines and when the container is started it auto-joins to to the cluster, we have only one such image / host machine.

Our ever-changing python project is submitted as a zip file along with the job and imports work transparently form there. Luckily we rarely need to re-create those slave images and we don't run jobs with conflicting requirements.

I'm not sure how applicable this in your scenario, especially since (in my understanding) some python libraries must be compiled.

NikoNyrh
  • 3,578
  • 2
  • 18
  • 32
  • I'd be very interested in seeing that "Spark slave script" if you're free to share it. – Stuart Jan 06 '16 at 19:30
  • That script refers to something like https://github.com/nikonyrh/docker-scripts/blob/master/startSparkSlave.sh, which is a wrapper for "docker run" command. Most importantly it sets SPARK_PUBLIC_DNS and uses "--net host" as mentioned at http://stackoverflow.com/a/32737662/3731823 – NikoNyrh Jan 06 '16 at 20:42