Convert a simple one line string to RDD in Spark

Question

I have a simple line:

line = "Hello, world"

I would like to convert it to an RDD with only one element. I have tried

sc.parallelize(line)

But it get:

sc.parallelize(line).collect()
['H', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']

Any ideas?

score 33 · Accepted Answer · answered Oct 02 '14 at 09:40

33

try using List as parameter:

sc.parallelize(List(line)).collect()

it returns

res1: Array[String] = Array(hello,world)

answered Oct 02 '14 at 09:40

michaeltang

6

Actually, using sc.parallelize([line]).collect() worked. – poiuytrez Oct 02 '14 at 09:46
1

in my env （ scala 2.10.3 spark-1.0.0 ) sc.parallelize([line]).collect() doesn't work – michaeltang Oct 02 '14 at 09:54
I forgot to specify that I am using the python api. – poiuytrez Oct 02 '14 at 10:00
@poiuytrez you also forgot to **accept** the answer! Nice question, +1 to both! – gsamaras Sep 06 '16 at 23:46
@michaeltang @poiuytrez Is this still the way to do it with `pyspark 3.0.0` – Learner Oct 25 '21 at 19:47

score 1 · Answer 2 · edited Nov 20 '20 at 14:54

1

The below code works fine in Python

sc.parallelize([line]).collect()
['Hello, world']

Here we are passing the parameter "line" as a list.

edited Nov 20 '20 at 14:54

Ruli

answered Dec 02 '18 at 12:00

Dhruv

score 0 · Answer 3 · edited Nov 20 '20 at 12:33

0

use the following code :

sc.parallelize(Seq(line))

edited Nov 20 '20 at 12:33

Suraj Rao

answered Nov 20 '20 at 12:32

vivek

it would be nicer if you add some explanation to your answer – mooga Nov 20 '20 at 14:43

3 Answers3