How to add partitions not in dynamic frame while writing data to S3 in AWS Glue script

Question

While writing the data to S3 using dynamic frame i want to use partitioning columns which are not in dynamic frame.

For example:

def write_date(outpath,year):
    glue_context.write_dynamic_frame.from_options(
        frame = projectedEvents,
        connection_type = "s3",    
        connection_options = {"path": outpath, "partitionKeys": [year]},
        format = "parquet")

Here year is a parameter which does not present in dynamic frame.

This code is failing with an error: 'partition column "2021" not found in schema'

How can I write data in S3 using my own partitions?

Basically I want to write in S3 path as "outpath/2021/<parquet_file>"

Robert Kossendey · Answer 1 · 2021-04-07T18:01:14.067

1

This would work:

projectedEvents = projectedEvents.withColumn('year', lit(2021))

def write_date(frame,outpath,year):
    glue_context.write_dynamic_frame.from_options(
        frame = frame,
        connection_type = "s3",    
        connection_options = {"path": outpath, "partitionKeys":[year]},
        format = "parquet")

write_date(projectedEvents, outpath, 'year')

I would suggest that you take another look into partitioning. It has to be a column of the data_frame.

edited Apr 07 '21 at 18:01

answered Apr 07 '21 at 16:55

Robert Kossendey

6,733
2
12
42

Hi Robert, actually year is a parameter to this write function – Beginner Apr 07 '21 at 17:20
Hi, as the error says, there is probably not a column named 2021 in the data frame. How do you call this function and what does your data frame look like? – Robert Kossendey Apr 07 '21 at 17:40
yes, 2021 is not a column presnt in data frame. Thats what i wanted to do- I wanted to add a partition column which is not present in data frame – Beginner Apr 07 '21 at 17:46
I updated my answer. I hope this is what you were looking for. – Robert Kossendey Apr 07 '21 at 18:01

How to add partitions not in dynamic frame while writing data to S3 in AWS Glue script

1 Answers1