iam trying to delete duplicate values by partition (dt), but iam getting error:
An error was encountered:
u'Cannot overwrite a path that is also being read from.;'
Query i'am using is:
query = "SELECT DISTINCT * FROM {} WHERE dt = '{}'".format(table_name, partition_date)
df = spark.sql(query)
df.createOrReplaceTempView("temp_table")
overwrite_query = "INSERT OVERWRITE TABLE {} PARTITION (dt) SELECT * FROM temp_table".format(table_name, partition_date)
Where iam I wrong ?
