0

All Databrew jobs that saves the result in S3 creates partitions of the resultant file. Due to this, we need to merge this partitioned files before we can use them in Excel. Is there any way by which I can stop the Databrew job from partitioning the resultant file?

Arijit
  • 25
  • 5

1 Answers1

1

When you go to create a DataBrew recipe job, there is a way to setup "File partitioning" option for each of your outputs. The option you are looking for is called "Single file output", select this option and run the job to create a single file per custom column based partitions out of your DataBrew recipe job.

enter image description here

AWS Documentation link: https://docs.aws.amazon.com/databrew/latest/dg/jobs.recipe.html#singlefileoutput

Nitya Sheth
  • 150
  • 9
  • Thank you @Nitya Sheth!! I think this feature was introduced recently because I checked this setting page a lot of times and never saw the option before. – Arijit Mar 04 '22 at 10:38
  • @Arijit - You are to the point. Here's the what's new announcement for the feature - https://aws.amazon.com/about-aws/whats-new/2022/02/aws-glue-databrew-output-files-recipe-jobs/ – Nitya Sheth Mar 29 '22 at 17:29