How to load typesafe configFactory from file on hdfs?

Question

I am using typesafe ConfigFactory to load the config into my scala application. I do not want to include the config files into my jar, but load them instead from an external hdfs filesystem. However, I cannot find a simple way to load the config from the fsDataInputStream object I get from hadoop:

//get HDFS file
val hadoopConfig: Configuration = sc.hadoopConfiguration
val fs: FileSystem = org.apache.hadoop.fs.FileSystem.get(hadoopConfig)
val file: FSDataInputStream = fs.open(new Path("hdfs://SOME_URL/application.conf"))
//read config from hdfs
val config: Config = ConfigFactory.load(file.readUTF())

However, this throws an EOFException. Is there an easy way to convert the FSDataInputStream object into the required java.io.File? I found Converting from FSDataInputStream to FileInputStream , but this would be pretty cumbersome for such a simple task.

score 6 · Accepted Answer · answered Jun 30 '16 at 13:39

6

Using ConfigFactory.parseReader should work (but I haven't tested it):

val reader = new InputStreamReader(file)
val config = try {
  ConfigFactory.parseReader(reader)
} finally {
  reader.close()
}

answered Jun 30 '16 at 13:39

Alexey Romanov

167,066
35
309
487

Works fine! Thank you – Malte Jun 30 '16 at 14:04
Can we include files inside typesafe config file from hdfs. `include file("hdfs://home//XYZ.conf")` – Pushpendra Jaiswal Aug 04 '16 at 06:19
I don't know. Why not try it? – Alexey Romanov Aug 04 '16 at 07:14

score 1 · Answer 2 · answered Feb 18 '19 at 22:31

1

You should be able to load .conf file in hdfs using the following code:

ConfigFactory.parseFile(new File("application.conf"));

Please keep in mind that the .conf file should be placed on the same directory as your app file (e.g. jar file in spark).

answered Feb 18 '19 at 22:31

Yohan Chung

519
1
6
15

score 1 · Answer 3 · edited Apr 06 '22 at 14:23

I could fix the issue with below code. Assume configPath is the path in HDFS location, where you have the .conf file available. for Ex:- hdfs://mount-point/abc/xyz/details.conf

import java.io.File
import com.typesafe.config._
import org.apache.hadoop.fs.{FileSystem, Path}
import java.io.InputStreamReader
val configPath = "hdfs://sparkmainserver:8020/file.conf"
val fs = FileSystem.get(new org.apache.hadoop.conf.Configuration())
val reader = new InputStreamReader(fs.open(new Path(configPath)))
val config: Config = ConfigFactory.parseReader(reader)

Then you can use config.getString("variable_name") to extract and use the variables/parameters. Prior to this you should have ConfigFactory sbt/Maven dependency in your pom file.

score 0 · Answer 4 · answered Apr 27 '18 at 14:25

Here is what I did with Spark application:

  /**
    * Load typesafe's configuration from hdfs file location
    * @param sparkContext
    * @param confHdfsFileLocation
    * @return
    */
  def loadHdfsConfig(sparkContext: SparkContext, confHdfsFileLocation: String) : Config = {
    // Array of 1 element (fileName, fileContent)
    val appConf: Array[(String, String)] = sparkContext.wholeTextFiles(confHdfsFileLocation).collect()
    val appConfStringContent = appConf(0)._2
    ConfigFactory.parseString(appConfStringContent)
  }

Now in the code, just use

val config = loadHdfsConfig(sparkContext, confHdfsFileLocation)
config.getString("key-here")

I hope it helps.

How to load typesafe configFactory from file on hdfs?

4 Answers4

Linked