How to clear driver and executor memory blocked data due to broadcast variable in spark scala during spark-submit job execution

Asked Feb 14 '22 at 12:54

Active Feb 14 '22 at 12:54

Viewed 505 times

I have series of spark jobs execution flow in single class as shown below:

SparkMainClass:

Job1 (Using 4 new dataframes in broadcast join)
Job2 (Using 3 new dataframes in broadcast join)
Job3 (Using 4 new dataframes in broadcast join)
Job4 (Using 2 new dataframes in broadcast join)

All the 4 jobs will execute one after another in sequence. So after each Job execution I want to clear dataframes used in broadcast join to save Driver and Executor memory, else I am encountering out-of-memory issue or suggesting to increase driver memory.

Is there any any way to clear broadcasted data from driver and executor memory dynamically. Like Job1 execution completed clear 4 df broadcast data and start Job2 execution and so on...

Please assist.

asked Feb 14 '22 at 12:54

Ku002

You don't need to clear this manually, spark will do this automatically for you. – Warren Zhu Feb 14 '22 at 15:46

How to clear driver and executor memory blocked data due to broadcast variable in spark scala during spark-submit job execution

0 Answers0