0

I am trying to create a simple working code in pyspark (Using Livy to access spark). I am using below link as reference. https://pylivy.readthedocs.io/en/latest/index.html

But it gives error says spark is not defined. 

from livy import LivySession

LIVY_URL = "<Livy_URL>"

with LivySession.create(LIVY_URL) as session:
    spark = SparkSession.builder.master("local").appName('file_filter').getOrCreate()
    df=spark.read.option("header","true").option("inferSchema","true").csv("file_path")
    #filter the rows with country = USA
    session.run("filtered = df.filter(df.Country == 'USA')")

    local_df = session.read("filtered")
    local_df.write.format("csv").save("output_file_path")

Error

    df=spark.read.option("header","true").option("inferSchema","true").csv("file_path")
NameError: name 'spark' is not defined

I think its not able to create the spark session.What change in the code is needed?

Codegator
  • 459
  • 7
  • 28
  • 1
    The link you cite as "reference" is about a wrapper around Livy REST API. **The whole point of Livy is that you don't run Spark locally**; the Livy session runs Spark remotely and automatically, based on the init params you give to Livy. So why did you paste some random code about that `SparkSession` in `local` mode??? – Samson Scharfrichter May 30 '20 at 20:51
  • 1
    To be more specific: you have some Python code that runs locally, and some Python code that is passed to Livy as a _script_ (i.e. text, between quotes, that is an argument to a method that sends the text to Livy server for execution). When you declare a variable locally, Livy is not aware of it. And vice versa. Sadly, you clearly don't understand anything about client/server architecture. – Samson Scharfrichter May 30 '20 at 20:55
  • Thanks Samsonfor the clarity.Can you please tell me then how to read a file and store it in dataframe using Livy and write the output back to a local file again. Once clarification here is do I need to send python code to Livy for execution or Pyspark code? – Codegator May 30 '20 at 21:36
  • I would appreciate if you can guide on how to change the above code to work. Thanks – Codegator May 30 '20 at 21:43
  • Sorry but S.O. is not the best place to find tutorials or guidance. – Samson Scharfrichter Jun 01 '20 at 13:27

0 Answers0