I am using Airflow to create a DAG to run a spark job. In the DAG I am setting the spark_conf (executor memory, driver memory, and maxResultSize) for my use case.
I received a GC memory error so, as I was reviewing the python code I noticed a spark session was being created where maxResultsSize was being set in the config.
My question is if this would override my spark_conf that I set in the Airflow DAG?
Asked
Active
Viewed 144 times
0
Prof. Falken
- 499
- 6
- 21
-
which operator are you using? – Jonathan Lam Aug 11 '22 at 15:07
-
@Jonathan I am using the `DatabricksSubmitRunOperator` – Prof. Falken Aug 11 '22 at 15:17