Stack Overflow
Stack Overflow

Questions tagged [livy]

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface

From http://livy.incubator.apache.org.

What is Apache Livy?

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or a RPC client library. Apache Livy also simplifies the interaction between Spark from application servers, thus enabling the use of Spark for interactive web/mobile applications. Additional features include:

  • Have long running Spark Contexts that can be used for multiple Spark jobs, by multiple clients
  • Share cached RDDs or Dataframes across multiple jobs and clients
  • Multiple Spark Contexts can be managed simultaneously, and the Spark Contexts run on the cluster (YARN/Mesos) instead of the Livy Server, for good fault tolerance and concurrency
  • Jobs can be submitted as precompiled jars, snippets of code or via java/scala client API
  • Ensure security via secure authenticated communication

References

288 questions
0
votes
1 answer

Spark rest url job submission in java

i am new to spark and would like to do rest url submission on a local machine, i was able to do a rest submission on a remote cluster.but everytime submitting it to remote cluster is a bit tedious , is there anyway i can submit the jar to spark rest…
Crazywolf
  • 41
  • 1
  • 1
  • 8
0
votes
1 answer

Livy Client - Spark_Conf.zip does not exist

I am trying to submit the application through livy batches through the Postman POST call. I do see the following error in the logs. But i am able to execute the commands in the interactive livy sessions through Curl. I checked the HADOOP_CONF_DIR…
pradz_stack
  • 176
  • 1
  • 11
0
votes
1 answer

(Zeppelin + Livy) SparkUI.appUIAddress(), something must be wrong

I'm trying to configure livy with Zeppelin following this docs: https://zeppelin.apache.org/docs/0.7.3/interpreter/livy.html However when I run: %livy.spark sc.version I got the following error: java.lang.RuntimeException: No result can be…
Yusata
  • 199
  • 1
  • 3
  • 16
0
votes
1 answer

Submitting Spark job in HDInsight cluster for secure enabled storage account

When we try to submit Spark job using ADF or Livy in HDInsight 3.6, Spark 2.1 with Secure transfer required "Enabled" in associated storage account. It is not submitting Spark job and we are unable to see Spark Job in YARN UI, but the same way we…
Sudeep Singh Thakur
  • 79
  • 1
  • 2
  • 10
0
votes
1 answer

livi server fails to connect to spark cluster with authentication enabled

I'am running a 3 node Spark 2.1.1 Standalone Cluster with spark.authenticate=true and the Livy Server fails to connect to it. Here is a part from my livy.conf livy.spark.authenticate = true livy.spark.master =…
bvasil
  • 1
  • 2
0
votes
2 answers

Submitting Python file in batch mode in Livy(without Hadoop installed)

i have made a simple python file which want to submit in Livy.Livy is currently running in local mode.Also I have mentioned following property in livy.conf file. Property name: livy.file.local-dir-whitelist, value "/usr/local/livy/scripts" My file…
Utkarsh Saraf
  • 475
  • 8
  • 31
0
votes
2 answers

IllegalStateException: Session is in state starting'

I installed apache spark and apache Livy in my system. When I am running a python code, it's giving error 'u'java.lang.IllegalStateException: Session is in state starting'' By default Apchy Livy run on port number 8998. My python code is import…
neha
  • 1,858
  • 5
  • 21
  • 35
0
votes
1 answer

Failure to find jar in repository when I build Livy using "mvn clean package"

I am a newbie of Livy and I build Livy as the official said. The command is as follows: git clone https://github.com/apache/incubator-livy.git cd incubator-livy mvn package Then I get error as follows： [ERROR] Plugin…
GOGO
  • 619
  • 2
  • 7
  • 17
0
votes
2 answers

Getting the version of Livy on the cluster

how to get what is the version of the livy I am using in the cluster? we have both livy client and server on cluster nodes but I need to know what is its version preferably on a horthonwork cluster.
Luckylukee
  • 575
  • 2
  • 9
  • 27
0
votes
1 answer

curl on Windows: "GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)"

I want to access a service that uses Kerberos authentication. The service admin gave me a spnego.service.keytab file for the correct Kerberos principal and realm. I can run kinit against that keytab file to get a Kerberos ticket successfully. But…
snark
  • 2,462
  • 3
  • 32
  • 63
0
votes
1 answer

Livy : Run spark job in yarn mode

Not able to run Spark job in yarn(cluster or client) mode through Livy, I am able to run spark job using Livy, but the jobs are running local mode and not visible on resource manager. I want to run these jobs in yarn-cluster mode. I am using…
Shantanu Sharma
  • 3,661
  • 1
  • 18
  • 39
0
votes
1 answer

Livy ERROR RSCClient: Failed to connect to context

I am running a Livy Client application in Scala ApI which Livy is 0.3.0 and Spark version is 1.6.0 on Yarn model. Here is the main code. def main(args: Array[String]): Unit = { var scalaClient: LivyScalaClient = null val livyurl =…
Jary zhen
  • 437
  • 6
  • 18
0
votes
1 answer

Spark word count job submitted using Livy Rest request goes into Dead state

I am running a spark word count program using Livy (Running it without Livy using spark-submit works fine). On submitting the job using Livy Rest request, it returns with id for the job as below: curl -X POST --data '{"file":…
Saurabh Rana
  • 168
  • 3
  • 22
0
votes
1 answer

"Internal Server Error: Session is in state starting" error while running sample Spark Livy job

"Internal Server Error: Session is in state starting" error while running sample Spark Livy job. I installed Livy for accessing apache-spark using REST API and started the Livy server. On running a sample spark Pi App(as giving on…
Saurabh Rana
  • 168
  • 3
  • 22
0
votes
1 answer

Livy return before the computation is over

Working with Livy on the Hortonworks Sandbox we found the problem that on some cases the Livy API provide a result for the statement before the actual work has ended as indicated by the Spark monitoring tool. In some cases Livy provide a result in…
yeiniel
  • 2,416
  • 15
  • 31
1 2 3
…
19
20