1

Why SparkHadoopUtil is not accessible here whereas is accessible in lower version of spark even though they are imported?

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.2
      /_/
         
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.deploy.SparkHadoopUtil

scala> import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.conf.Configuration

scala> 

scala> 

scala>  val hadoopConf: Configuration = SparkHadoopUtil.get.conf
<console>:25: error: object SparkHadoopUtil in package deploy cannot be accessed in package org.apache.spark.deploy
        val hadoopConf: Configuration = SparkHadoopUtil.get.conf
                                        ^

scala>
mck
  • 40,932
  • 13
  • 35
  • 50
supernatural
  • 1,107
  • 11
  • 34

1 Answers1

2

That's because the SparkHadoopUtil class has been changed to a private class in Spark 3. Here's the difference between the source of Spark 2.4 and Spark 3.0.

Spark 2.4:

@DeveloperApi
class SparkHadoopUtil extends Logging {

Spark 3.0:

private[spark] class SparkHadoopUtil extends Logging {
mck
  • 40,932
  • 13
  • 35
  • 50
  • Thanks @mck , but in that case is there a work around for it as i was using it for getting the file system as `FileSystem.get(SparkHadoopUtil.get.conf)` – supernatural Mar 22 '21 at 07:10
  • See [this question](https://stackoverflow.com/q/32380272). You can try: `import org.apache.hadoop.conf.Configuration` `import org.apache.hadoop.fs.FileSystem` `val conf = new Configuration()` `val fs = FileSystem.get(conf)` – mck Mar 22 '21 at 07:27
  • `new Configuration()` is not completly correct for the hadoop conf config by `--conf` @mck – Wechar Yu Mar 16 '23 at 10:51