How to pass Spark executor parameters to DBT?

Question

I am using DBT with spark-livy-adapter. And I want to pass spark-conf properties like --num_executors via DBT-cli command: like this

dbt run --select my_model --spark_conf "num-executors":5

Please me help me on this

Answer 1

I have added the livy_session_parameters in parameter in the profile.yml file and added the value as a variable then I'm passing the executor number as a variable and it's working fine.

My profile.yml file look like this

dbt_project:
  outputs:
   my_spark_livy:
     livy_session_parameters:
       spark.executor.instances: "{{ var('num_executor') }}"
  target: my_spark_livy

DBT cli command:

dbt run --select my_model --vars '{"num_executor" :"10"}'