there's a hudi table that is written as parquet file in s3, I ma trying to query it using athena, firstly it worked fine, then when I try to add a column and try to query it again I get this error:
GENERIC_INTERNAL_ERROR: Field new_test2 not found in log schema. Query cannot proceed! Derived Schema Fields:
despite the fact that when I try to query the same table using spark.sql it works fine. I don't know why this is happening as far as I understand that athena can handle schema changes, so why it's mentioning that the col doesn't exist? and also if I tried to alter the table to add the col, we get an error of duplicate col which means it can see it. ps: this error happens in athena engine v3, but when I set athena to automatic it works fine.
