How to alter the column datatype of the managed delta table using pyspark?

Question

How to alter the column datatype based on the input parameter using pyspark

from pyspark.sql.types import IntegerType,BooleanType,DateType
from pyspark.sql.functions import col
Column_Name="EFFECTIVE_DATE"
df=spark.sql(f"select * from TableA")
df.withColumn({Column_Name},col({Column_Name}).cast(DateType())

from where this input comes? widgets? or something else? – Alex Ott Nov 10 '21 at 07:08 — Alex Ott, Nov 10 '21 at 07:08
datafactory input parameter – Venkatesh Nov 10 '21 at 07:18 — Venkatesh, Nov 10 '21 at 07:18

score 0 · Answer 1 · answered Nov 10 '21 at 12:18

0

input come always through widget:

input = dbutils.widgets.get("X")

so assume that:

input == "int"

then instead of using IntegerType() function you can just use string "int" type as cast so:

df.withColumn({Column_Name},col({Column_Name}).cast(input)

answered Nov 10 '21 at 12:18

Hubert Dudek

1,666
1
13
21

How to alter the column datatype of the managed delta table using pyspark?

1 Answers1