How can i use Spark Context?

Question

I assign the value as sc = pyspark.SparkContext(). It run and doesnt respond for so long on jupyter notebook as asteric sign appears and doesnt show any error or so.

I tried sc = SparkContext()

import pyspark
import os
from pyspark import SparkContext, SparkConf
sc = pyspark.SparkContext()  # At this part it don't respond
from pyspark.sql import SQLContext
sqlc = SQLContext(sc)

It should go on.

which version of spark are you using? – mythic May 30 '19 at 08:49 — mythic, May 30 '19 at 08:49
I am using pyspark 2.3.4 – Syed Mustafa Nadeem May 30 '19 at 13:47 — Syed Mustafa Nadeem, May 30 '19 at 13:47

score 1 · Answer 1 · answered May 31 '19 at 06:31

For Python,

from pyspark import SparkContext
sc = SparkContext(appName = "test")

But since you're working on pyspark version 2+ , you dont need to initialize spark context. You can create a spark session and directly work on it.

SPARK 2.0.0 onwards, SparkSession provides a single point of entry to interact with underlying Spark functionality and allows programming Spark with DataFrame and Dataset APIs. All the functionality available with sparkContext are also available in sparkSession.

In order to use APIs of SQL, HIVE, and Streaming, no need to create separate contexts as sparkSession includes all the APIs.

To configure a spark session,

session = SparkSession.builder.getOrCreate()

If you use sparkSession, you wont have to create sparkContext, sqlContext separately and spark session will work as an entrypoint for both. — mythic, May 31 '19 at 06:33

score -1 · Answer 2 · answered May 29 '19 at 13:46

-1

Try the following import: from pyspark import * After that you can use it like so:

sc = SparkContext()

answered May 29 '19 at 13:46

Ballo Adam

105
1
8

1

[Do not do `import *`](https://stackoverflow.com/questions/49481363/pyspark-nonetype-object-has-no-attribute-jvm-error/55711135#55711135). – pault May 29 '19 at 14:17
What should I do? – Syed Mustafa Nadeem May 29 '19 at 14:35

How can i use Spark Context?

2 Answers2