Stack Overflow
Stack Overflow

Questions tagged [livy]

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface

From http://livy.incubator.apache.org.

What is Apache Livy?

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or a RPC client library. Apache Livy also simplifies the interaction between Spark from application servers, thus enabling the use of Spark for interactive web/mobile applications. Additional features include:

  • Have long running Spark Contexts that can be used for multiple Spark jobs, by multiple clients
  • Share cached RDDs or Dataframes across multiple jobs and clients
  • Multiple Spark Contexts can be managed simultaneously, and the Spark Contexts run on the cluster (YARN/Mesos) instead of the Livy Server, for good fault tolerance and concurrency
  • Jobs can be submitted as precompiled jars, snippets of code or via java/scala client API
  • Ensure security via secure authenticated communication

References

288 questions
1
vote
1 answer

How to enable CORS in Apache Livy?

I am trying to send POST requests to Livy in javascript from my website using fetch (). However, CORS is enabled by default and the request fails. On setting the mode to 'no-cors', the livy server sends an empty string as a response ( I cannot…
user3266985
  • 13
  • 5
1
vote
1 answer

Issue in adding dependencies from local Repository into Apache Livy Interpreter for Zeppelin

How to add dependencies to the Apache Livy Interpreter in Apache Zeppelin? I tried adding the way, I did for standard Spark Interpreter inside Zeppelin, it did not work I tried adding it on livy.spark.jars.packages, it did not work. All these…
Harvinder Singh
  • 681
  • 7
  • 20
1
vote
1 answer

Unable to connect to remote server when trying to call Invoke-RestMethod from Azure Runbook

I am encountering this error when trying to call Invoke-RestMethod from Powershell and Powershell Workflow scripts on Azure Runbook. Unable to connect to the remote server (Unable to connect to the remote server (A connection attempt failed because…
Katy McClintic
  • 11
  • 1
  • 3
1
vote
0 answers

Submit R script to Spark via Livy on remote standalone server

I set as standalone a Ubuntu server with Spark 2.2.0 up and running. My aim is to allow several users (clients) to connect to this server and develop locally (from their own computer) thanks to RStudio, some code, which has to be executed on…
NathJ
  • 49
  • 5
1
vote
1 answer

How to build the Livy client on Windows?

I want to write a Java app on Windows that uses Livy's Java API to talk to an Apache Livy server that's hosted remotely. The docs say: Add the Livy client dependency to your application's POM: org.apache.livy …
snark
  • 2,462
  • 3
  • 32
  • 63
1
vote
1 answer

How to set proxy user in Livy Job submit through its Java API

I am using Livy's Java API to submit a spark job on YARN on my cluster. Currently the jobs are being submitted as 'livy' user, but I want to submit the job as a proxy user from Livy. It is possible to do this by sending POST request to the Livy…
Lokesh Yadav
  • 958
  • 2
  • 9
  • 20
1
vote
2 answers

livy - server limit number of jobs

I am looking for some configuration in order to limit my livy server to not use more than X cores or number of jobs, I saw that someone already asked this in the Jira issues (https://issues.cloudera.org/browse/LIVY-74 ), and he got answered that…
Ehud Lev
  • 2,461
  • 26
  • 38
1
vote
1 answer

How do I submit a jAR with dependencies to Livy?

I'm using Livy on HDInsight to submit jobs to a Spark cluster. I have my code written and compiled into a JAR, but it has multiple dependencies, some of which are from a custom repository. How do I get Livy to resolve these dependencies by itself? I…
SiddharthaRT
  • 2,217
  • 4
  • 20
  • 28
1
vote
2 answers

spark.executor.cores can't take effect while using livy

I'm using livy-server-0.2 to run spark job, however, I can't change the default setting for spark.executor.cores, it can't take effect while the other settings can. It always use 1 core to start an executor. yarn 11893 11889 6 21:08 ? …
Ron.Lin
  • 11
  • 1
  • 2
1
vote
1 answer

Maven enforcer plugin 1.4.1 or one of its dependencies cannot be resolved

I am trying to build livy-server on one of the datanodes of our hadoop cluster which has all the prerequisites like Spark 1.6.2, hadoop 2.4.2 already set up. I am following the instrucctions as described in the link…
Srinivas Santhanam
  • 23
  • 1
  • 7
1
vote
1 answer

Python Requests Post request fails when connecting to a Kerberized Hadoop cluster with Livy

I'm trying to connect to a kerberized hadoop cluster via Livy to execute Spark code. The requests call im making is as below. kerberos_auth = HTTPKerberosAuth(mutual_authentication=REQUIRED, force_preemptive=True) r = requests.post(host +…
vivek
  • 27
  • 5
1
vote
1 answer

How to configure Livy / spark for Zeppelin?

I have successfully configured Zeppelin with Spark. However, I want to use multi-tenancy, and for that I want to configure Zeppelin with Livy and Spark. For Livy, I provided the following two paths export…
Dan
  • 115
  • 1
  • 10
0
votes
0 answers

Spark Cannot execute python script references module imports

I'm facing challenges when using Spark in conjunction with Livy. Specifically, I run into exceptions when my jobs require imports from different modules: If I don't use the pyfiles attribute: I receive a "file not found" error. If I utilize the…
Udemytur
  • 79
  • 1
  • 5
0
votes
0 answers

AWS MER: livy and spark: not working sometimes. Strange behaviour

I am using livy to start a spark job inside aws EMR and see sometimes it keeps failing as below. The following is from the livy Log 23/08/15 01:27:09 ERROR SparkYarnApp: Error whiling refreshing YARN state java.lang.IllegalStateException: No YARN…
Santhosh
  • 9,965
  • 20
  • 103
  • 243
0
votes
0 answers

spark and livy: state running to dead, how to troubleshoot

I am running a spark jon in an EMR cluster using livy response= requests.post("http://"+host + ':8998/batches', data=json.dumps(data), headers=headers) and now trying to poll to see the spark job is done in side a while loop check…
Santhosh
  • 9,965
  • 20
  • 103
  • 243
1 2 3
…
19 20