1

Scenario :

  • I have spark cluster and
    I also want to use Livy.
  • I am new about Livy

Problem :

  • I built my spark cluster by using docker swarm and
    I will also create a service for Livy.

  • Can Livy communicate with external spark master and send a job to external spark master?
    If it is ok, which configuration need to be done?
    Or Livy should be installed on spark master node?

Abhishek Kumar
  • 2,501
  • 10
  • 25
ugur
  • 400
  • 6
  • 20

1 Answers1

1

I think is a little late, but I hope this help you.

sorry for my english, but I am mexican, you can use docker to send jobs via livy, but also you can use livy to send jobs throw Livy REST API.

The livy server can be outside of the spark cluster, you only need to send a conf file to livy that points to you spark cluster.

It looks you are running spark standalone, easist way to configure livy to work is that livy lives on spark master node, if you already have YARN on your cluster machines, you can install livy on any node and run spark application in yarn-cluster or yarn-client mode.

César Ramos
  • 11
  • 1
  • As you said "The livy server can be outside of the spark cluster, you only need to send a conf file to livy that points to you spark cluster." So what are the the steps to install livy server outside the cluster. Please give details about which config files and properties of config file need to change to point to spark cluster – Bector Sep 24 '19 at 10:01
  • @Bector did you install livy outside spark cluster? – Avik Aggarwal Dec 01 '20 at 12:08