hadoop distributed copy overwrite not working

Question

I am trying to use the org.apache.hadoop.tools.DistCp class to copy some files over into a S3 bucket. However overwrite functionality is not working in spite of explicitly setting the overwrite flag to true

Copying works fine but it does not overwrite if there are existing files. The copy mapper skips those files. I have explicitly set the "overwrite" option to true.

import com.typesafe.scalalogging.LazyLogging
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.tools.{DistCp, DistCpOptions}
import org.apache.hadoop.util.ToolRunner
import scala.collection.JavaConverters._

object  distcptest extends  App with LazyLogging {


  def copytoS3( hdfsSrcFilePathStr: String, s3DestPathStr: String) = {
    val hdfsSrcPathList = List(new Path(hdfsSrcFilePathStr))
    val s3DestPath = new Path(s3DestPathStr)
    val distcpOpt = new DistCpOptions(hdfsSrcPathList.asJava, s3DestPath)

    // Overwriting is not working inspite of explicitly setting it to true.
    distcpOpt.setOverwrite(true)

    val conf: Configuration = new Configuration()
    conf.set("fs.s3n.awsSecretAccessKey", "secret key")
    conf.set("fs.s3n.awsAccessKeyId", "access key")
    conf.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")

    val distCp: DistCp = new DistCp(conf, distcpOpt)
    val filepaths: Array[String] = Array(hdfsSrcFilePathStr, s3DestPathStr)

    try {
      val distCp_result = ToolRunner.run(distCp, filepaths)
      if (distCp_result != 0) {
        logger.error(s"DistCP has failed with - error code = $distCp_result")
      }
    }
    catch {
      case e: Exception => {
        e.printStackTrace()
      }
    }
  }

  copytoS3("hdfs://abc/pqr", "s3n://xyz/wst")
}

score 0 · Answer 1 · answered May 14 '20 at 07:09

I think the problem is you called ToolRunner.run(distCp, filepaths).

If you check the source code of DistCp, in run method will overwrite inputOptions, so the DistCpOptions passed to constructor will not work.

  @Override
  public int run(String[] argv) {
    ...

    try {
      inputOptions = (OptionsParser.parse(argv));
      ...
    } catch (Throwable e) {
      ...
    }
    ...
  }

hadoop distributed copy overwrite not working

1 Answers1