0

Am running a spark application which will use configuration parameters from a file.

File:- Spark.conf

username=ankush
password=ankush
host=https://
port=22
outputDirectory=/home/ankush/data/

How to use this file at runtime. Instead of restarted the job when we make changes to the configuration file how to make the job pick the file at runtime dynamically.

I tried to use it with spark-submit using --conf spark.yarn.appMasterEnv but this is looking for a variable not for a file.

Is there any possibility to implement this.

Thanks in advance for your help.

Community
  • 1
  • 1
ankush reddy
  • 481
  • 1
  • 5
  • 28

1 Answers1

0

You can keep the variables in conf/spark-defaults.conf file.

Ref: https://spark.apache.org/docs/latest/configuration.html#dynamically-loading-spark-properties

jdprasad
  • 367
  • 4
  • 15
  • Thank You for pointing me to that. I have one question will this be able to pick the properties at runtime. Let's say I have set all the properties in the conf/spark-defaults.conf after submitting the job using spark-submit if I change the properties in conf/spark-defaults.conf will this effect the already running spark application or do I have to restart the application to pick the changes from conf/spark-defaults.conf. – ankush reddy Sep 01 '17 at 17:08
  • Assuming it's about same one long running spark job, when job starts it should read a certain value of property, when the job is midway is should read the same property but updated ones (some other program or user updated those properties while the program was executing) and may be another set of values at another point of time during job execution of same long running job ?Is that what you want? – jdprasad Sep 02 '17 at 14:59