RDD Partitions Issue

Question

Reading a csv file using below code

rdd= (sc.sparkContext
       .textFile("./test123.txt")
       .flatMap(lambda line: line.split("\n\r"))
       .map(lambda x: x.split("|"))
      )

On running the above code spark just creates one partition(on my local machine) , wanted to understand why is that ? The below display 1

rdd.getNumPartitions()

I want to parallelize this operation so can be run on the cluster. For the work to be distributed the rdd should have more then one partition (that's my understanding ) so that task can be sent to other nodes. Any light on this ?

https://stackoverflow.com/questions/27299923/how-to-load-local-file-in-sc-textfile-instead-of-hdfs — thebluephantom, May 30 '21 at 07:10

score 0 · Answer 1 · answered May 31 '21 at 15:40

"The textFile method also takes an optional second argument for controlling the number of partitions of the file. By default, Spark creates one partition for each block of the file (blocks being 128MB by default in HDFS), but you can also ask for a higher number of partitions by passing a larger value. Note that you cannot have fewer partitions than blocks."

please try for example:

.textFile("./test123.txt", 2)

RDD Partitions Issue

1 Answers1