2

I am trying to setup Livy and Spark on Docker.

Right now I have a local setup of Spark as spark-shell is running fine on my Windows CMD, and I created an image of Livy (i.e. tobilg/livy) on Docker as Livy doesn't work on Windows directly. That image of Livy also working well but they don't seem to connect with each other, So, I have to create a container of Spark as well.

Please help me out here.

Thanks in advance.

Divya Arya
  • 439
  • 5
  • 22
  • Please provide details about how you are trying to connect the services. Creating another container just to solve the connectivity problem might not be the best solution unless containerization is your final goal. – r3mus n0x Jul 22 '18 at 11:14
  • Connection, basically Livy should simply call spark-submit in local mode, that's it. So, what should be the feasible solution to do that on Windows? – Divya Arya Jul 22 '18 at 11:18
  • The problem is that it's not local. Since Livy is in the container it's the same as if it has been on the other host. – r3mus n0x Jul 22 '18 at 11:26
  • Okay, then what's the other way? – Divya Arya Jul 22 '18 at 11:28
  • 1
    It seems that you already have Spark in your Livy image. Can you use it instead of your local installation? Alternatively you can try to mount your local installation into container as a volume using `-v :/usr/local/spark` parameter of `docker run` but I'm not sure it will work on Windows. – r3mus n0x Jul 22 '18 at 11:58
  • If spark already in Livy image, that is great, I can use it but I have no idea how? – Divya Arya Jul 22 '18 at 12:01
  • There is nothing in README file, about using spark. – Divya Arya Jul 22 '18 at 12:03
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/176511/discussion-between-r3mus-n0x-and-divine). – r3mus n0x Jul 22 '18 at 12:05

0 Answers0