Is there any way to change data type inside struct column in a table?
Example:
emp_details: struct
emp_id: integer
emp_name: string
If emp_details is a column in a table which is of strict type and inside it emp_id and emp_name is present and I want to change emp_id to string.
Yes you can. You should explicitly cast the column and build the new emp_details using the casted column. Once you create the desired dataframe you can overwrite the table in Databricks to store it with the desired schema.
This should look something like this:
# For code readability, let's first create the correct casted column
original_df_with_casted_column_df = original_df.withColumn("casted_emp_id", col("emp_details.emp_id").cast("string"))
# We generate the new struct field using the original emp_name column and the newly created column after renaming it.
final_df = original_df.select(struct(col("casted_emp_id").alias("emp_id"), col("emp_name")).alias("emp_details"))
