Stack Overflow
Stack Overflow

Questions tagged [livy]

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface

From http://livy.incubator.apache.org.

What is Apache Livy?

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or a RPC client library. Apache Livy also simplifies the interaction between Spark from application servers, thus enabling the use of Spark for interactive web/mobile applications. Additional features include:

  • Have long running Spark Contexts that can be used for multiple Spark jobs, by multiple clients
  • Share cached RDDs or Dataframes across multiple jobs and clients
  • Multiple Spark Contexts can be managed simultaneously, and the Spark Contexts run on the cluster (YARN/Mesos) instead of the Livy Server, for good fault tolerance and concurrency
  • Jobs can be submitted as precompiled jars, snippets of code or via java/scala client API
  • Ensure security via secure authenticated communication

References

288 questions
1
vote
1 answer

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails) livy 0.7.1 CDH 6.3.2(enable kerberos) in livy.conf livy.server.auth.type = kerberos livy.server.auth.kerberos.principal =…
ighack
  • 31
  • 4
1
vote
0 answers

Livy timeout when using s3 jars

When using REST request to open livy sessions's with jars from s3 I got an error, I checked the s3 login and endpoint, they are good, I dont understand why this doenst work livy_1 | Creating Interactive session 2: [owner: null, request:…
Yassine
  • 123
  • 3
1
vote
0 answers

Failing to submit spark job through Livy programmatic client api

I'm trying to use submit simple word count program which was implemented with java8, spark 2.4.7, and submitted through Livy 0.7. I have three containers which runs on my local computer. One for Spark driver, one for Spark worker and one for Livy. I…
Oded
  • 336
  • 1
  • 3
  • 17
1
vote
0 answers

How can I scala 2.12 on Livy 0.7

Is there any option to run scala 2.12 spark 3.1 by Livy 0.7? Is there any estimation of Livy 0.8 release?
Ilan K
  • 11
  • 1
1
vote
2 answers

Submitting Spark Job to Livy (in EMR) from Airflow (using airflow Livy operator)

I am trying to schedule a job in EMR using airflow livy operator. Here is the example code I followed. The issue here is... nowhere Livy connection string (Host name & Port) is specified. How do I provide the Livy Server host name & port for the…
Raj
  • 2,368
  • 6
  • 34
  • 52
1
vote
0 answers

Livy starts throwing java.lang.IllegalStateException: failed to create a child event loop

I am working with EMR 6.2.0 and livy 0.7.0 I have a worker which keeps launching spark sessions and polling for its completion. This runs fine for a while but then suddenly Livy starts throwing java.lang.IllegalStateException: failed to create a…
Dhiraj Soni
  • 33
  • 6
1
vote
0 answers

Submit Livy Pyspark job with python driver in remote server

I am trying to run a POST /batches command through Livy with the following configurations: { 'files': ['some.pex'], 'file': 'script.py', 'conf': {'spark.pyspark.python': '/directory/some.pex'} } The command works as long as the…
sdev001
  • 31
  • 3
1
vote
1 answer

Getting import error while executing statements via livy sessions with EMR

I am trying to post statements to livy session with EMR 6.1.0. But i am unable to import the class(to my custom jar) which i am trying to execute. Statement I am trying to post to a livy session - import com.path.to.Compactor …
Dhiraj Soni
  • 33
  • 6
1
vote
0 answers

Unable to set environment variables in Spark using livy and sparkmagic

Scenario : I have setup a spark cluster on my kubernetes environment : Livy Pod for submission of jobs Spark Master Pod Spark Worker Pod for execution What I want to achieve is as follows: I have a jupyter notebook with a Pyspark kernel as a pod…
Devesh
  • 21
  • 3
1
vote
0 answers

SparkMagic: Reuse SparkSession in other Jupyter notebook

I setup a Hadoop cluster with Spark 2.4, on my client I installed Jupyter notebook and the SparkMagic module. Everything seems to work fine within the same notebook. But now I want to reuse my SparkSession in another Jupyter notebook, for example I…
D. Müller
  • 3,336
  • 4
  • 36
  • 84
1
vote
1 answer

Fetching the Spark Yarn log from Azure HDInsight

Currently through LIVY I am Posting/submitting spark jobs to Azure HDInsight Cluster. After job finishes I am looking into Spark History Server for yarn logs. Livy log for each spark job is not providing yarn logs. Can we Fetch the Spark Yarn log…
samrat1
  • 29
  • 1
  • 1
  • 6
1
vote
0 answers

How to integrate Spark, Livy and Nifi to execute spark code from jar

Is there a way, we can execute Spark code(package in jar) from Nifi, using Livy? I can see in Nifi that using ExecuteSparkInteractive, we can submit custom code which can be run in spark cluster using livy. but what i want is , pass name of the jar…
GPopat
  • 445
  • 4
  • 14
1
vote
1 answer

Configure Livy log4j properties on EMR Cluster

I am using rest Apis provided by livy to submit spark jobs on EMR cluster.I am able to overwrite some of the livy properties in livy-conf file using below json in configuration while creating cluster - [{'classification': 'livy-conf','Properties':…
gagana
  • 73
  • 1
  • 1
  • 6
1
vote
0 answers

spark master not able to handle concurrent job submission (AWS EMR)

I am running AWS EMR cluster (emr-5.30.1, Spark 2.4.5, Livy 0.7.0). My service pass jobs to livy and livy execute "spark-submit" in cluster mode to submit jobs to yarn. spark master is 8 core, 16GB machine. I see dead jobs when ~15-20 jobs are…
Manjeet Rulhania
  • 81
  • 1
  • 6
1
vote
0 answers

Livy session gets corrupted

we are using a Apache Livy server to communicate with a spark cluster. We have noticed that some statements when executed error out and afterwards livy session becomes unusable. For example we are using fbprophet for timeseries models. fbprophet and…
user3104078
  • 107
  • 1
  • 7
1 2 3
…
19 20