Stack Overflow
Stack Overflow

Questions tagged [livy]

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface

From http://livy.incubator.apache.org.

What is Apache Livy?

Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or a RPC client library. Apache Livy also simplifies the interaction between Spark from application servers, thus enabling the use of Spark for interactive web/mobile applications. Additional features include:

  • Have long running Spark Contexts that can be used for multiple Spark jobs, by multiple clients
  • Share cached RDDs or Dataframes across multiple jobs and clients
  • Multiple Spark Contexts can be managed simultaneously, and the Spark Contexts run on the cluster (YARN/Mesos) instead of the Livy Server, for good fault tolerance and concurrency
  • Jobs can be submitted as precompiled jars, snippets of code or via java/scala client API
  • Ensure security via secure authenticated communication

References

288 questions
4
votes
0 answers

Run Livy Job in a Kerberos-enabled Hadoop Cluster

I created an example Livy (Spark) application using the com.cloudera.livy.Job class for calculating an approximate value for Pi (Source: https://github.com/cloudera/livy#using-the-programmatic-api), exported as jar file to e.g.…
D. Müller
  • 3,336
  • 4
  • 36
  • 84
4
votes
2 answers

Configure external jars with HDI Jupyter Spark (Scala) notebook

I have an external custom jar that I would like to use with Azure HDInsight Jupyter notebooks; the Jupyter notebooks in HDI use Spark Magic and Livy. Within the first cell of the notebook, I'm trying to use the jars configuration: %%configure…
Denny Lee
  • 3,154
  • 1
  • 20
  • 33
3
votes
4 answers

How to rebuild apache Livy with scala 2.12

I'm using Spark 3.1.1 which uses Scala 2.12, and the pre-built Livy downloaded from here uses Scala 2.11 (one could find the folder named repl_2.11-jars/ after unzip). Referred to the comment made by Aliaksandr Sasnouskikh, Livy needs to be rebuilt…
Jeff Shih
  • 45
  • 1
  • 6
3
votes
0 answers

Livy start new session

I have a problem when I want to create a new session by livy the session dead after their creation, I have installed Livy , spark 3.0.0 , scala 1.12.10 and python 3.7.I followed these steps: step 01 : start Livy server ./livy-server start step…
zekri sidi mohamed hicham
  • 57
  • 1
  • 10
3
votes
1 answer

Running Custom Java Class in PySpark on EMR

I am attempting to utilize the Cerner Bunsen package for FHIR processing in PySpark on an AWS EMR, specifically the Bundles class and it's methods. I am creating the spark session using the Apache Livy API, def create_spark_session(master_dns,…
user1983682
  • 208
  • 6
  • 20
3
votes
2 answers

Read spark stdout from driverLogUrl through livy batch API

Livy has a batch log endpoint: GET /batches/{batchId}/log, pointed out in How to pull Spark jobs client logs submitted using Apache Livy batches POST method using AirFlow As far as I can tell, these logs are the livy logs and not the spark driver…
Nevermore
  • 7,141
  • 5
  • 42
  • 64
3
votes
1 answer

Cannot send Python dependencies to Spark on EMR via Livy

I searched and viewed a looot of articles online and finally found a possible solution in: I can't seem to get --py-files on Spark to work I followed the most voted answer in that link but it is not working for me. And that topic is 3 years old and…
Z.Wei
  • 3,658
  • 2
  • 17
  • 28
3
votes
0 answers

Problem connecting Livy to EMR via sagemaker

I have followed this tutorial: https://aws.amazon.com/fr/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/ in order to be able to run pyspark code on EMR via apache-livy. I have only made some little change so…
Elie Ladias
  • 111
  • 2
  • 16
3
votes
2 answers

livy curl request error for Kerberos Cloudera Hadoop

Configured livy server on kerberized CDH 5.10.x and its running fine on port 8998, but curl request giving below error, curl --negotiate -u : http://xxxxxxx:8998/sessions Error 403 HTTP ERROR: 403 Problem accessing /sessions. Reason: …
Ajay Kharade
  • 1,469
  • 1
  • 17
  • 31
3
votes
1 answer

How to submit batch jar Spark jobs by livy Programmatic API

I want to submit batch jar Spark jobs using livy Programmatic API, like using rest API batches, I have the json data { "className": "org.apache.spark.examples.SparkPi", "queue": "default", "name": "SparkPi by Livy", "proxyUser":…
寒江雪
  • 71
  • 5
3
votes
1 answer

Spark application gets KILLED abruptly in EMR after 1 hour and livy session expires.What is the cause& solution?

I am using JupyterHub on AWS EMR cluster. I am using EMR version 5.16 I submitted a spark application using a pyspark3 notebook. My application is trying to write 1TB data to s3. I am using autoscaling feature of the EMR to scale us the task…
Hiranya Deka
  • 242
  • 6
  • 15
3
votes
1 answer

Matplotlib does not plot when using Apache Livy interpreter on Zeppelin

I am unable to use matplotlib with Apache Livy 0.5 on Zeppelin 0.8. Paragraph does not display the plot. Please check the sample code below: %livy.pyspark import matplotlib.pyplot as plt plt.plot([1, 2, 3]) I am getting the following…
Harvinder Singh
  • 681
  • 7
  • 20
3
votes
2 answers

Jupyter as Zeppelin replacement: multi-lingual Spark

My team is trying to transition from Zeppelin to Jupyter for an application we've built, because Jupyter seems to have more momentum, more opportunities for customization, and be generally more flexible. However, there are a couple of things…
Ken Williams
  • 22,756
  • 10
  • 85
  • 147
3
votes
0 answers

Spark session could not be created from hue server

I am trying to create a spark session from hue server which has been configured with livy server to communicate with kerberos enabled spark. I am getting the below error when trying to start spark scala session using Query editor. But i can able to…
Karthik Sridhar
  • 239
  • 1
  • 6
3
votes
2 answers

Issue in accessing zeppelin context in Apache Livy Interpreter for Zeppelin

Is it possible to access zeppelin context (z), in Livy Interpreter for Zeppelin? If yes, then how to access it. If no, does that mean zeppelin visualization in Livy is limited to python libraries. How does one exchange variables between scala and…
Harvinder Singh
  • 681
  • 7
  • 20
1 2
3
…
19 20