In my case , the source filed is having blank/"" empty string. in Bronze layer when i am reading it i have used .option("nullValue", "null") and i am using autoloader in raw to bronze. The value is written in bronze delta table as expected which is "" blank. But in silver layer the value is written as Null again. from bronze to silver i am using spark.sql to read from bronze table and keeping it in a dataframe and then writing to silver delta table.
bronze_refined_df = spark.sql(f"select {final_cols} from {catalog_name}.{bronze_database_name}.{tableName} where {loadDateColumnName} > (SELECT nvl(max(lh_load_date), to_timestamp('1900-01-01 00:00:00.000')) from {catalog_name}.{silver_database_name}.{tableName})")
can anyone hep me with if i want to write the blank string as blank in silver layer also how to approach.
