1

We're trying to run ALTER DDL statements on existing Athena tables, previously created through a regular Java SDK StartQueryExecutionRequest without saving versions, so we don't run into the TABLE_VERSION Glue limit (see Glue limits link below). We had run our application for a while, unaware that previous versions were all being stored, and we ran into a hard limit in our AWS account. Specifically, we're adding partitions and also updating avro schemas programatically using the AWS Java SDK version 2 (2.10.66, if that matters).

It looks like we need to enable an option called SkipArchive using the Glue UpdateTable request to disable this previous versions functionality. However, the AWS documentation, while extensive, is all over the place, doesn't have clear examples, and I can't figure out how to simply run a basic UpdateTableRequest.

Anybody has any example/links that we could look at that details how to run this command through the Java API?

Doc listing the limits of table versions per table and per account in Glue - https://docs.aws.amazon.com/general/latest/gr/glue.html

https://docs.aws.amazon.com/glue/latest/webapi/API_UpdateTable.html#API_UpdateTable_RequestSyntax

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/glue/AWSGlue.html#updateTable-com.amazonaws.services.glue.model.UpdateTableRequest-

mjuarez
  • 16,372
  • 11
  • 56
  • 73
  • I'm running into the same issue. Did you manage to find it? If so, kindly share with us on this thread. @mjuarez – RMu Jan 14 '21 at 14:36

1 Answers1

0

The Answer might be bit late but here is a update_table request example for reference :

import boto3

glue=boto3.client('glue')

#List A Specific Table from Glue Catalog

""" 
paginator = glue.get_paginator('get_table_versions')

response_iterator  = paginator.paginate(DatabaseName='etl_framework',TableName='etl_master_withskiparchive')

for page in response_iterator:
    print(page)
"""

#Sample glue.create_table method

#glue.create_table(DatabaseName='etl_framework',TableInput={'Name':'etl_master_20221013','StorageDescriptor':{'Columns':[{'Name':'id','Type':'bigint','Comment':''},{'Name':'groupid','Type':'bigint','Comment':''},{'Name':'tableName','Type':'string','Comment':''},{'Name':'schemaName','Type':'string','Comment':''},{'Name':'source','Type':'string','Comment':''},{'Name':'loadType','Type':'string','Comment':''},{'Name':'incrementalcolumn','Type':'string','Comment':''},{'Name':'active','Type':'bigint','Comment':''},{'Name':'createdate','Type':'date','Comment':''},{'Name':'startdatetime','Type':'timestamp','Comment':''},{'Name':'enddatetime','Type':'timestamp','Comment':''},{'Name':'sql_query','Type':'string','Comment':''},{'Name':'partition_column','Type':'string','Comment':''},{'Name':'priority','Type':'bigint','Comment':''},{'Name':'pk_columns','Type':'string','Comment':''},{'Name':'merge_query','Type':'string','Comment':''},{'Name':'bucket_Name','Type':'string','Comment':''},{'Name':'offset_value','Type':'bigint','Comment':''},{'Name':'audit','Type':'bigint','Comment':''}],'Location':'s3://pb-4-0-datalake-raw-layer/iceberg/etl_master_20221013','InputFormat':'','OutputFormat':'','Compressed':False,'NumberOfBuckets':0,'SerdeInfo':{},'BucketColumns':[],'SortColumns':[],'SkewedInfo':{},'StoredAsSubDirectories':False},'Parameters':{'metadata_location':'s3://pb-4-0-datalake-raw-layer/iceberg/etl_master_20221013/metadata/','table_type':'ICEBERG'},'TableType': 'EXTERNAL_TABLE'})

#Sample glue.update_table method

glue.update_table(DatabaseName='etl_framework',TableInput={'Name':'etl_master_20221011','StorageDescriptor':{'Columns':[{'Name': 'id', 'Type': 'bigint', 'Parameters': {'iceberg.field.current': 'true', 'iceberg.field.id': '1', 'iceberg.field.optional': 'true'}}],'Location':'s3://pb-4-0-datalake-raw-layer/iceberg/etl_master_20221010','InputFormat':'','OutputFormat':'','Compressed':False,'NumberOfBuckets':0,'SerdeInfo':{},'BucketColumns':[],'SortColumns':[],'SkewedInfo':{},'StoredAsSubDirectories':False},'Parameters':{'metadata_location':'s3://pb-4-0-datalake-raw-layer/iceberg/etl_master_20221012/metadata/','table_type':'ICEBERG'},'TableType': 'EXTERNAL_TABLE'},SkipArchive=True)
Satheesh V
  • 96
  • 7