Stack Overflow
Stack Overflow

Questions tagged [livy]

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface

From http://livy.incubator.apache.org.

What is Apache Livy?

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or a RPC client library. Apache Livy also simplifies the interaction between Spark from application servers, thus enabling the use of Spark for interactive web/mobile applications. Additional features include:

  • Have long running Spark Contexts that can be used for multiple Spark jobs, by multiple clients
  • Share cached RDDs or Dataframes across multiple jobs and clients
  • Multiple Spark Contexts can be managed simultaneously, and the Spark Contexts run on the cluster (YARN/Mesos) instead of the Livy Server, for good fault tolerance and concurrency
  • Jobs can be submitted as precompiled jars, snippets of code or via java/scala client API
  • Ensure security via secure authenticated communication

References

288 questions
0
votes
1 answer

How to submit a Job to Livy (under Knox) with Programmatic API?

This code from Livy Docs worked fine: LivyClient client = new LivyClientBuilder() .setURI(new URI("http://sample.ru:8999")) .build(); String piJar =…
Makrushin Evgenii
  • 953
  • 2
  • 9
  • 20
0
votes
1 answer

NoClassDefFoundError: org/apache/solr/client/solrj/io/stream/expr/StreamExpressionParameter in Solr and Spark Livy

I am trying to read data stored in Solr through the spark, but I am not able to show the execession below. However I am passing the following drivers in the Livy configuration. I have tried other driver versions, but to no avail. My Spark is version…
Cícero Moura
  • 23
  • 5
0
votes
1 answer

Pass a pySpark script in Livy Session statement

I understand livy session statement intakes code statements like the below example. data = { 'code': textwrap.dedent(""" import random NUM_SAMPLES = 100000 def sample(p): x, y = random.random(), random.random() return 1 if x*x + y*y < 1 else…
Shubzumt
  • 143
  • 1
  • 12
0
votes
1 answer

How rewrite spark scala code to use it in apache livy

i rewrite this code: import org.apache.spark.sql.SparkSession object SimpleApp { def main(args: Array[String]) { val logFile = "file:///root/spark/README.md" val spark = SparkSession.builder.appName("Simple Application").getOrCreate() …
seyyed heydar javadi
  • 103
  • 1
0
votes
1 answer

getting error while using livybatch operator in Airflow , DAG getting crashed

Can someone help me on this while using livybatchoperator in Airflow , below is my code ... apart from that what's other way to run spark job in airflow except spark operator, spark is installed on different machine in my case. I'm getting this…
Hitesh Sharma
  • 1
0
votes
1 answer

Calling Apache Livy REST API from Java

We use Apache Livy to read data from object storage in a Spark cluster. Apache Livy API: POST http://server:port/sessions/{sessionId}/statements Request Payload: {"code":" val df = spark.read.option(\"header\",…
Jobinesh
  • 87
  • 2
  • 10
0
votes
0 answers

Create spark session for processing files using Livy : Anaconda Spyder

I am trying to create a simple working code in pyspark (Using Livy to access spark). I am using below link as reference. https://pylivy.readthedocs.io/en/latest/index.html But it gives error says spark is not defined. from livy import…
Codegator
  • 459
  • 7
  • 28
0
votes
0 answers

Livy spark interactive session

I'm trying to create spark interactive session with livy .and I need to add a lib like a jar that I mi in the hdfs (see my code ) . but the session is dead and the log is below. code : client =…
zekri sidi mohamed hicham
  • 57
  • 1
  • 10
0
votes
1 answer

LDAP Authentication on Apache Livy

How do I enable LDAP Authentication on Apache Livy? I have seen the following resources but I haven't found anything that documents how to utilize the…
Joshua Villanueva
  • 179
  • 1
  • 5
  • 13
0
votes
0 answers

NIFI LIVYSESSION CONTROLLER SERVICE GIVING ERROR : READ TIMEOUT

I want to submit spark jobs from NIFI using Executivespark interactive processor, for that I am setting up livysession controller service. The ip and port names are correct, when I try to connect it's throwing error as shown in the attachment can…
devanshmbi
  • 3
  • 3
0
votes
1 answer

Airflow Livy jobs subission on AWS EMR

We are using Airflow to schedule our jobs on EMR and currently we want to use apache Livy to submit Spark jobs via Airflow I need more guidance on below : Which Airflow-Livy operator we should use for python 3+ pyspark and scala jobs. I have seen…
user10437665
  • 95
  • 2
  • 9
0
votes
0 answers

Initiating Spark session from Livy in Kerberised Cluster HDP

I am trying to start an interactive spark session through livy in kerberised spark cluster. I have gone through basic articles and have configured services. The spark session is getting launched in yarn mode. Issue comes when yarn is trying to start…
Aviral Kumar
  • 814
  • 1
  • 15
  • 40
0
votes
1 answer

Livy Batch API - not able to get the jar

I am trying to use Livy Batch API to invoke my Spark application. curl -X POST -d '{"file": "file:///path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar","className": "App","args": ["abc"],"jars": [],"driverMemory": "16g","executorMemory":…
Chris
  • 23
  • 7
0
votes
1 answer

Apache Livy - Rest API

My Requirement : From web application trigger a Spark job in Yarn and display the result back in web page. The spark job accepts few arguments and computes a DataSet with values that needs to be returned to web application. After doing some browsing…
Chris
  • 23
  • 7
0
votes
1 answer

Where should Apache Livy be installed in cluster

We want to use apache Livy so that we can invoke spark job from restapi. So do we need to install Livy server on name node or edge node. What is the best practice. Our spark fat jar will reside in NFS path.
Sparkuser191
  • 23
  • 5
1 2 3
…
19 20