Stack Overflow
Stack Overflow

Questions tagged [livy]

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface

From http://livy.incubator.apache.org.

What is Apache Livy?

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or a RPC client library. Apache Livy also simplifies the interaction between Spark from application servers, thus enabling the use of Spark for interactive web/mobile applications. Additional features include:

  • Have long running Spark Contexts that can be used for multiple Spark jobs, by multiple clients
  • Share cached RDDs or Dataframes across multiple jobs and clients
  • Multiple Spark Contexts can be managed simultaneously, and the Spark Contexts run on the cluster (YARN/Mesos) instead of the Livy Server, for good fault tolerance and concurrency
  • Jobs can be submitted as precompiled jars, snippets of code or via java/scala client API
  • Ensure security via secure authenticated communication

References

288 questions
1
vote
1 answer

Uploading jar to Apache Livy interactive session

Using Amazon emr-5.30.1 with Livy 0.7 and Spark 2.4.5 We are willing to use Apache Livy as a REST Service for spark. The mode we want to work with is session and not batch. Trying to upload a jar to the session (by the formal API) using: curl -X…
user2992499
  • 11
  • 3
1
vote
1 answer

execute batch job on livy within python virtualenv created already in server

I am trying to execute a python file using batch job on livy Url: http://:8998/batches body: { "file": "hdfs://localhost:9000/sample.py" } i want to execute this file within python virtualenv which is already created on server. I…
Akshata Dasar
  • 11
  • 2
1
vote
1 answer

Livy create session dead

I added to my spark config a package (in spark-default.conf) but when I create a new session with livy it causes me a problem (see the error below) and the session and death . ps: when I remove this package all work fine . 20/05/04 00:17:35 WARN…
zekri sidi mohamed hicham
  • 57
  • 1
  • 10
1
vote
0 answers

Error while creating a spark session with apache Livy

I am trying to create a spark session using apache livy, but I am not able to create a session on the spark cluster and I am not getting any helpful logs. Is there a way I could find the actual reason for this error. Any help would be…
AS-Sher
  • 384
  • 1
  • 4
  • 18
1
vote
0 answers

How to run multiple instances of Apache Livy server with different ports?

I am trying to run two Apache Livy instances on the same machine using different ports and pointing to the same Spark cluster. But when starting the second Livy service I am getting this error: livy-server running as process 24222. Stop it…
Vikneswaran S J
  • 58
  • 7
1
vote
1 answer

How to access Livy on a Kerberized cluster (or disable Kerberos for Livy)?

I created an AWS EMR cluster with Kerberos enabled, and I am able to authenticate Hadoop with my Kerberos keytab using the Java hadoop.conf authentication method. However, I cannot access Livy as I keep running into this error: HTTP ERROR:…
JYCH
  • 61
  • 7
1
vote
0 answers

Spark submit not respecting spark.driver.supervise=true set in spark-defaults.conf

What issue I faced? I specified spark.driver.supervise=true in spark-defaults.conf ${SPARK_HOME}/bin/spark-submit script did not pick this configuration. It started job/driver with spark.driver.supervise=false PS: So far I have only found…
surajs21
  • 128
  • 1
  • 8
1
vote
1 answer

How can I see executor logs in Livy running in kubernetes

I am having a stable/spark Helm deployment running on Kubernetes and submitting the job through Livy. curl -X POST --data '{"className": "LogBundleConfigFetcher", "file": "http:///aliceparser_2.11-19.12.09.jar" ,"args": [""]}' -H "Content-Type:…
Sumit G
  • 436
  • 8
  • 21
1
vote
1 answer

How can we install Apache Livy outside spark cluster?

My apology if somewhere I made a mistake in my language. I want to install the Apache Livy server on a node(VM Instance) outside the Spark cluster. How can I do this so that LivyServer should point to the Spark cluster? I have downloaded and…
Umesh Gaikwad
  • 301
  • 3
  • 14
1
vote
0 answers

Apache livy error - "Unable to connect to server" spark://[hostname]:7077

I'm trying to submit a Spark batch job using Apache Livy. I am running my Spark cluster and Livy service on Openshift. the Spark cluster is in standalone cluster mode, Livy runs spark in cluster deploy mode. I've configured Livy to the Spark cluster…
Matar Peles
  • 11
  • 1
1
vote
0 answers

How to authenticate users sending a job through Livy on a Dataproc Cluster without Kerberos

We are trying to restrict users from sending a job to data proc cluster through livy server. By default livy submits job as user 'livy' when impersonation is disabled. Even after using impersonation, it says livy is not allowed to impersonate user…
user11631629
1
vote
1 answer

How do I determine the Apache Livy's "allowed maximum active session count"?

Let's say, I want to keep creating a sessions for every Spark job that is submitted to the YARN. Every connection has a unique user, who keeps polling the status and post statements to a session. How do I calculate, at any given time, Livy can have,…
Anandkumar
  • 1,338
  • 13
  • 15
1
vote
1 answer

issue to connect jupyter sparkmagic kernel to kerberized livy server

please help if you have any idea: I am trying to connect kerberized Hortonworks hadoop clusters livy server with jupyter, I have 401 error when connecting Is it possible to connect sparkmagic to kerberized spark livy server? if it is then I think I…
Shota Akhalaia
  • 11
  • 2
1
vote
2 answers

Emr Notebook - Session isn't active

I'm using EMR Notebooks with pyspark and livy. I'm reading the data from s3 which is in parquet format and string into a pyspark dataframe. there are approx. 2 million rows. when i do a join operation. I am getting 400 session isn't active. for…
Kandarp
  • 126
  • 5
1
vote
1 answer

Is livy available for dse versin 6.04

Hi I am trying to install livy in dse cluster version 6.0.4. I have copied the livy foder in dse and trying to execute livy server: dse exec livy-server But getting error unknown command exec I have checked the dse documentation for this version…
Kumar Harsh
  • 423
  • 5
  • 26
1 2 3
…
19 20