AWS Glue jobs not writing to S3

Question

I have just been playing around with Glue but have yet to get it to successfully create a new table in an existing S3 bucket. The job will execute without error but there is never any output in S3.

Here's what the auto generated code is:

glueContext.write_dynamic_frame.from_options(frame = applymapping1, 
connection_type = "s3", connection_options = {"path": 
"s3://glueoutput/output/"}, format = "json", transformation_ctx = 
"datasink2")

Have tried all variations of this - with name of file (that doesn't exist yet), in root folder of bucket, trailing slash and without. The role being used has full access to S3. Tried creating buckets in different regions. No file is ever created though. Again console says its successful.

Are you using an auto-generated script as a starting point or creating your own from scratch? That looks fine so I'm thinking that some other piece of code is broken. — MadCityDev, Sep 25 '17 at 14:57

score 5 · Answer 1 · answered Mar 06 '21 at 11:25

As @Drellgor suggests in his comment to the previous answer, make sure you disabled "Job Bookmarks" unless you definitely don't want to process old files.

From the documentation:

"AWS Glue tracks data that has already been processed during a previous run of an ETL job by persisting state information from the job run. This persisted state information is called a job bookmark. Job bookmarks help AWS Glue maintain state information and prevent the reprocessing of old data."

score 3 · Answer 2 · answered Sep 25 '17 at 22:31

3

your code is correct, just verify if there is any data at all in applymapping1 DF? you check with this command : applymapping1.toDF().show()

answered Sep 25 '17 at 22:31

letstry

83
1
7

2

This proved that the Dynamic Frame had data by displaying it in the log. I also disabled bookmarking. Doing these two things resolved the issue for me. – Drellgor Apr 10 '18 at 16:46

score 3 · Answer 3 · answered Mar 08 '21 at 11:15

You need to edit your IAM role. You should define your IAM role can write and also read from the S3.

Go to your AWS console
Go to IAM
Policies
Edit Policy
Add following Put and Delete Object for S3 in addition to get object.
Then save

Be sure you are running AWS Glue with the IAM Role you edited. Good Luck.

"Effect": "Allow",
        "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject"

score 1 · Answer 4 · answered Feb 07 '22 at 20:48

Had the same issue. After a few days of what seemed like Glue jobs randomly write to s3 and sometimes don't I found this thread. @Sinan Erdem suggestion resolved my issue.

From the aws docs:

Job bookmarks are used to track the source data that has already been processed, preventing the reprocessing of old data. Job bookmarks can be used with JDBC data sources and some Amazon Simple Storage Service (Amazon S3) sources. Job bookmarks are tied to jobs. If you delete a job, then its job bookmark is also deleted.

You can rewind your job bookmarks for your Glue Spark ETL jobs to any previous job run, which allows your job to reprocess the data. If you want to reprocess all the data using the same job, you can reset the job bookmark

Also found this: How to rewind Job Bookmarks on Glue Spark ETL job?

AWS Glue jobs not writing to S3

4 Answers4