2

Can we have an AWS Glue job scheduled to perform MSCK repair commands so that the metadata for newly added partition gets added to Glue Catalog?

Can Glue ETL script perform MSCK REPAIR TABLE command without calling Athena?

Sumedh
  • 21
  • 1
  • 4

2 Answers2

0

This is achieved by Glue Crawlers. If you create a crawler it will update the table based on new fields and add new partitions.

Ricardo Mayerhofer
  • 2,121
  • 20
  • 22
  • I have migrated Hive metastore tables to Glue Data catalog hence the the tables have different properties. When I run crawlers , it creates entirely new tables with added partitions. – Sumedh Apr 11 '18 at 07:39
0

You can call batch_create_partition() API to do it. It doesn't require expensive operations like MSCK REPAIR TABLE or re-crawling. Below is my detailed answer with code sample -

https://stackoverflow.com/a/52239022/2414855

conetfun
  • 1,605
  • 4
  • 17
  • 38