Stack Overflow
Stack Overflow

Questions tagged [livy]

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface

From http://livy.incubator.apache.org.

What is Apache Livy?

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or a RPC client library. Apache Livy also simplifies the interaction between Spark from application servers, thus enabling the use of Spark for interactive web/mobile applications. Additional features include:

  • Have long running Spark Contexts that can be used for multiple Spark jobs, by multiple clients
  • Share cached RDDs or Dataframes across multiple jobs and clients
  • Multiple Spark Contexts can be managed simultaneously, and the Spark Contexts run on the cluster (YARN/Mesos) instead of the Livy Server, for good fault tolerance and concurrency
  • Jobs can be submitted as precompiled jars, snippets of code or via java/scala client API
  • Ensure security via secure authenticated communication

References

288 questions
2
votes
1 answer

Sending python script with arguments to Apache Livy

I am trying to send python script with arguments to Apache Livy. Already I have a code which sends content of file (file_content) to Livy using following python code: data = { 'code': file_content } headers = {'Content-Type':…
flyingdonkey
  • 45
  • 4
2
votes
1 answer

Not able to create local directory in EMR by user 'livy'

I am submitting a pyspark job into EMR cluster from a AWS Step Function through apache livy. The pyspark job has Unix shell commands being fired. Within test.py, subprocess.call(' echo $USER', shell=True, stdout=None, stderr=None) subprocess.call('…
Parijat Bose
  • 380
  • 1
  • 6
  • 22
2
votes
1 answer

Reflection error while creating spark session through Livy

I am creating a spark session through LivyClient. The code for creating the session is - var client: LivyClient = null try { client = new LivyClientBuilder() .setURI(new URI("http://127.0.0.1:4141")) .build(); val pi =…
Naina Jain
  • 21
  • 1
2
votes
2 answers

Timeout error: Error with 400 StatusCode: "requirement failed: Session isn't active."

I'm using Zeppelin v0.7.3 notebook to run Pyspark scripts. In one paragraph, I am running script to write data from dataframe to a parquet file in a Blob folder. File is partitioned per country. Number of rows of dataframe is 99,452,829. When the…
user10644448
  • 21
  • 1
  • 4
2
votes
0 answers

Connecting to a Remote Cloudera Spark Cluster using Sparklyr with the method Livy

Not able to connect to a Remote spark cluster using Sparklyr Livy Method. config <- livy_config(username="", password="") sc <- spark_connect(master = "
", method = "livy", config = config) I am getting an error: Error…
vn17
  • 21
  • 2
2
votes
0 answers

Livy connection thru sparklyr in RStudio, slower than connection thru sparkmagic

I am using sparklyr(0.8.4) in RStudio to connect to my remote spark environment through livy, and noticed that it took about 3~5 minutes for sparklyr to establish a session. sc <- sparklyr::spark_connect(master="https://myremotelivy",…
ying_in_sj
  • 21
  • 5
2
votes
1 answer

Cached dataset not used in Livy

I have an application that submits jobs using livy. In the same livy session, various jobs are submitted. At times these jobs might be working on similar datasets, and so I want to reuse data from one job to another. I am caching the dataset in the…
user9024779
  • 87
  • 8
2
votes
0 answers

Apache Livy Security

We are trying to build a setup where we have a server that submits jobs of different users to the Livy server via the REST API. We established a kerberos server to authenticate against livy. But we want to prohibit the users to access a different…
Harun Zengin
  • 43
  • 6
2
votes
0 answers

How to setup Apache Livy and Spark in Docker?

I am trying to setup Livy and Spark on Docker. Right now I have a local setup of Spark as spark-shell is running fine on my Windows CMD, and I created an image of Livy (i.e. tobilg/livy) on Docker as Livy doesn't work on Windows directly. That…
Divya Arya
  • 439
  • 5
  • 22
2
votes
2 answers

Read file created in HDFS with Livy

I am using Livy to run the wordcount example by creating jar file which is working perfectly fine and writing output in HDFS. Now I want to get the result back to my HTML page. I am using Spark scala, sbt, HDFS and Livy. The GET/batches REST API…
Divya Arya
  • 439
  • 5
  • 22
2
votes
2 answers

How to make a HDInsight/Spark cluster shrink when idle?

We use Spark 2.2 on Azure HDInsight for ad hoc exploration and batch jobs. The jobs should run ok on a 5x medium VM cluster. They are 1. notebooks (Zeppelin with Livy.spark2 magics) 2. compiled jars being run with Livy. I have to remember to scale…
Iain
  • 1,797
  • 1
  • 20
  • 38
2
votes
0 answers

Livy and Elasticsearch-Spark: Multiple ES-Hadoop versions detected

I'm trying to read from elasticsearch in a livy job using the elastisearch-spark jar. When I upload the jar to a livy client(like the example here) I get this error and I'm not sure how to parse it. Caused by: java.lang.RuntimeException:…
Exuro
  • 229
  • 3
  • 15
2
votes
3 answers

How to set Spark configuration properties using Apache Livy?

I don't know how to pass SparkSession parameters programmatically when submitting Spark job to Apache Livy: This is the Test Spark job: class Test extends Job[Int]{ override def call(jc: JobContext): Int = { val spark = jc.sparkSession() …
Markus
  • 3,562
  • 12
  • 48
  • 85
2
votes
0 answers

Livy always run on local mode

I am trying to run Pyspark (or Spark) job via Livy server with "spark.master=yarn". What I have done: 1) In spark-defaults.conf: spark.master yarn spark.submit.deployMode client 2) In livy.conf: livy.spark.master = yarn livy.spark.deployMode =…
Grzegorzg
  • 659
  • 1
  • 4
  • 17
2
votes
0 answers

Zeppelin livy.sparkr interpreter failed with multiple line script

In my zeppelin (0.7.3) livy2.sparkr interpreter, scripts that more than 1 line couldn't work. Then I tried this in R: > parse(text="hello<-function(name){\r\n sprintf(\"Hello %s\", name);}") Error in parse(text = "hello<-function(name){\r\n …
Rox
  • 141
  • 5
1 2 3
…
19 20