What is the difference between spark-shell and pyspark, in terms of the language we use to write codes?

Question

I wrote

 a = sc.parallelize([1,2,3])

in spark-shell and got error

error: illegal start of simple expression
       a = sc.parallelize([1,2,3])
                      ^

but when I wrote this in PySpark, it worked.

What's the difference between the two?

That's the difference between Scala and Python. – Bala Sep 17 '21 at 08:14 — Bala, Sep 17 '21 at 08:14
okay. so scala works in spark shell and python in pyspark? – Abhay Sep 17 '21 at 10:23 — Abhay, Sep 17 '21 at 10:23
Correct. When using spark-shell, it expects Scala syntax – Bala Sep 17 '21 at 10:31 — Bala, Sep 17 '21 at 10:31

score 1 · Answer 1 · answered Sep 17 '21 at 08:21

1

You need to use scala to utilize spark-shell. In this case, it would be something like this

 val a = sc.parallelize(Seq(1, 2, 3))

answered Sep 17 '21 at 08:21

Filip

1 Answers1