3

I have an application that will get job configurations informed thru files (likely to have dozens of them each describing a particular job to be done by the app, preferably being flexible about where they are located), and for that we chose HOCON format and typesafe libs.

It works well in development loading from the resources folder, but the intent is to pass the path to these configuration files at runtime thru a parameter while calling spark-submit (spark-submit ... pathToFile ...).

But reading the path to the file fails with this error:

val jConfig = ConfigFactory.load(path)

com.typesafe.config.ConfigException$Missing: system properties: No configuration setting found for key 'configuration'

How can I read config files from other locations than /resources?

Solution

As per @hagarwal tip, the following worked

val input = Source.fromFile(path,"UTF8").mkString
val jConfig = ConfigFactory.parseString(input)
Welsige
  • 149
  • 1
  • 10

1 Answers1

2

Store config in HDFS/S3/ADFS/Local file system, pass Configuration file path to the program as argument, read config file as input stream (or file) form the configuration file path.

val confPath = args(0) //Configuration file path
val stream = Client.getObject(confPath) //Client -> HDFS/S3/ADFS/Local
val configString = Source.fromInputStream(stream.asInstanceOf[InputStream]).mkString
val config = ConfigFactory.parseString(configString)
hagarwal
  • 1,153
  • 11
  • 27
  • Will try to implement this, but what package the object "Client" comes from? – Welsige Oct 03 '19 at 15:06
  • The client will come from the package HDFS/S3/ADFS/local depending upon where you store the configuration file. Best will be if you stored the conf file on distributed file system HDFS/S3/ADFS instead of the local file system. – hagarwal Oct 03 '19 at 15:23
  • I could not find an import that provided me with the Client class and getObject method. But from your tip to use parseString, I was able to make i twork with this: val input = Source.fromFile(path,"UTF8").mkString Thanks – Welsige Oct 03 '19 at 16:46