37

I have a RDD and I want to convert it to pandas dataframe. I know that to convert and RDD to a normal dataframe we can do

df = rdd1.toDF()

But I want to convert the RDD to pandas dataframe and not a normal dataframe. How can I do it?

user2966197
  • 2,793
  • 10
  • 45
  • 77

2 Answers2

47

You can use function toPandas():

Returns the contents of this DataFrame as Pandas pandas.DataFrame.

This is only available if Pandas is installed and available.

>>> df.toPandas()  
   age   name
0    2  Alice
1    5    Bob
Community
  • 1
  • 1
jezrael
  • 822,522
  • 95
  • 1,334
  • 1,252
17

You'll have to use a Spark DataFrame as an intermediary step between your RDD and the desired Pandas DataFrame.

For example, let's say I have a text file, flights.csv, that has been read in to an RDD:

flights = sc.textFile('flights.csv')

You can check the type:

type(flights)
<class 'pyspark.rdd.RDD'>

If you just use toPandas() on the RDD, it won't work. Depending on the format of the objects in your RDD, some processing may be necessary to go to a Spark DataFrame first. In the case of this example, this code does the job:

# RDD to Spark DataFrame
sparkDF = flights.map(lambda x: str(x)).map(lambda w: w.split(',')).toDF()

#Spark DataFrame to Pandas DataFrame
pdsDF = sparkDF.toPandas()

You can check the type:

type(pdsDF)
<class 'pandas.core.frame.DataFrame'>
chhantyal
  • 11,874
  • 7
  • 51
  • 77
RKD314
  • 1,125
  • 3
  • 13
  • 18
  • 3
    I think `pdsDF = sparkDF.toPandas` is missing the () to actually call the method. It should be: `pdsDF = sparkDF.toPandas()` – learn2day Jun 26 '17 at 21:17
  • What is the difference between toDF() and toPandas()? – jtlz2 Mar 19 '19 at 10:21
  • toDF() converts an RDD to a Spark DataFrame, and toPandas() converts a Spark DataFrame to a Pandas DataFrame. The two kinds of DataFrames are different types. – RKD314 Jun 27 '19 at 18:57