Firstly, I read a csv file and save it to parquet format dataframe. After that, when I read this parquet data file with my new custom schema, returned dataframe has new schema but all NULL content. Comparing with old schema, I only change column names, all column type(string type) remains the same. enter image description here
As far as I know, schema metada will be created along while we write to parquet file. Does that mean we're not able change schema after data saved into parquet? I was expecting I can change atleast column name when reading
