0

I have successfully configured PySpark kernel in jupyter notebook, I also installed SparkMagic. When I try to use the below command:

%%sql
SELECT DepDelay, ArrDelay FROM flightData

it starts working and suddenly Spark stops throwing the below error:

An error was encountered: Invalid status code '400' from http://localhost:8998/sessions/0/statements/4 with error payload: {"msg":"requirement failed: Session isn't active."}

you can find the full log file here to download and take a look. https://drive.google.com/open?id=1lvYqQBUCiIFp4lz3aVnzMgBNd9fzqJiz

Appreciate your help. Many thanks in advance

M. Wadi
  • 111
  • 1
  • 1
  • 9
  • before running this line, are you sure that a connection is established, if not try spark.applicationID – Sarath Chandra Vema Oct 24 '19 at 16:06
  • everything else works, for example: the next lines, so i guess the application is running. data.createOrReplaceTempView("flightData") spark.sql("SELECT DayOfWeek, AVG(ArrDelay) AS AvgDelay FROM flightData GROUP BY DayOfWeek ORDER BY DayOfWeek").show() – M. Wadi Oct 24 '19 at 22:15

1 Answers1

0

Try in a single line like this

%sql SELECTDepDelay, ArrDelay FROM flightData
Sarath Chandra Vema
  • 792
  • 1
  • 6
  • 13
  • This did now work, however, I loaded a smaller number of rows into the data frame and the command "%%sql" worked, so I guess it's related to my PC's resources, are there any configurations that could be done? – M. Wadi Oct 25 '19 at 12:05