convert dataframe to list of rows pyspark glue

Question

How can I convert my dataframe df to a list of rows?

Code

df = glueContext.create_dynamic_frame_from_options(
    connection_type = "s3",
    connection_options = {"paths": ["s3://data/tmp1/file.csv"]},
    format = "csv",
)
df = df.toDF()
list = df.values.tolist()

Error

dataframe has no attribute values

do you mean **df = df.toPandas()** and then **liste = df.values.tolist()** ? — sygma, Jan 16 '20 at 18:55
Isn't this a duplicate of https://stackoverflow.com/q/34817549/11301900 ? — AMC, Jan 16 '20 at 22:53

E. Zeytinci · Accepted Answer · 2020-01-16T19:34:27.503

2

IMHO, you can use toPandas(),

df = glueContext.create_dynamic_frame_from_options(
    connection_type="s3", 
    connection_options={"paths": ["s3://data/tmp1/file.csv"]}, 
    format="csv")

df = df.toPandas()
liste = df.values.tolist()

edited Jan 16 '20 at 19:34

answered Jan 16 '20 at 19:10

E. Zeytinci

2,642
1
20
37

score 0 · Answer 2 · answered Jan 20 '20 at 06:03

In glue, you may use DyanamicFrame.map() method (https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html#aws-glue-api-crawler-pyspark-extensions-dynamic-frame-map)

df.map(to_list)
def to_list(rec):
       rec["list"] = [rec["col1"], rec["col2"] ]
       del rec["col1"]
       del rec["col2"]

convert dataframe to list of rows pyspark glue

2 Answers2