0

I want the updated value in the target's column to be the sum of source value + target value

example:

%scala 
import org.apache.spark.sql.functions._
import io.delta.tables._ 

// Create example delta table 
val dept = Seq(("Finance",10), ("Marketing",20),("Sales",30), ("IT",40) )
val deptColumns = Seq("dept_name","dept_emp_count")
val deptDF = dept.toDF(deptColumns:_*) 
deptDF.write.format("delta").mode("overwrite").saveAsTable("dept_table")

//create example stage dataframee
val staged_df = spark.sql("select * from dept_table").withColumn("dept_emp_count", lit(1))

//How to do this merge?
DeltaTable.forName(spark, "dept_table").as("events")
  .merge(staged_df.as("updates"), "events.dept_name = updates.dept_name")
.whenMatched()
.updateExpr(Map(                                      
  "dept_emp_count" -> lit("events.dept_emp_count") + lit("updates.dept_emp_count"))) // How do I write this line?
.execute()
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Gadam
  • 2,674
  • 8
  • 37
  • 56

1 Answers1

1

The value in that update Map is the SQL expression, so instead of the lit("events.dept_emp_count") + lit("updates.dept_emp_count") you just need to write "events.dept_emp_count + updates.dept_emp_count"

Alex Ott
  • 80,552
  • 8
  • 87
  • 132