aws glue drop partition using spark sql

Question

drop partition using spark sql frm glue metadata is throwing issues while same code works in hive shell.

**Hive shell**
hive> alter table prc_db.detl_stg drop IF  EXISTS partition(prc_name="dq") ;
OK
Time taken: 1.013 seconds


**spark shell**
spark.sql(''' alter table prc_db.detl_stg drop IF  EXISTS partition(prc_name="dq") ''') ;

Error message:
py4j.protocol.Py4JJavaError: An error occurred while calling o60.sql.
: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: InvalidObjectException(message:Unsupported expression (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException

Have you tried using delete_partition instead https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.delete_partition ? — Prabhakar Reddy, May 07 '21 at 07:23

score 0 · Answer 1 · answered May 14 '21 at 21:23

If you are planning to drop the partition(Glue Catalogue) from spark shell then, you have to add the "" in the partition name. ex: spark.sql(""" alter table reinvent.newtesttable drop partition ( part= " 'part2' " ) """) for me it was spark.sql(''' alter table prc_db.detl_stg drop IF EXISTS partition(prc_name="'dq'") ''')

aws glue drop partition using spark sql

1 Answers1