0

I'm doing some testing to integrate the Delta lake format into AWS Athena, currently, I have some delta tables already in Athena by manually creating the symlink format manifest.

I was reading a recent article that allows you to create AWS Delta Glue Crawlers to crawl the metadata found within the _delta_log folder to stop using the symlink format manifest, but after some testing I found out that it is not necessary to run the crawler every time a change is made to the delta table (if only, it's just necessary to run the crawler to create the table within the Glue catalog).

My question is: Are the crawlers really necessary to keep reading the metadata?

lealvcon
  • 178
  • 8
  • 1
    As I understand, I haven't worked with this just read. That you can use the crawler to create the metadata table. The crawler can detect changes to the schema and update for you in your glue table. If where wont be any changes to the structure the crawler is not needed after the table is created. – Erik Asplund Feb 17 '23 at 07:52

0 Answers0