0

I need to save the data in both HDFS and AWS S3 at a time. i have used below command but only first given path is working.

sqoop import -D mapreduce.job.name=XXX-D mapred.job.queue.name=XX -Dhadoop.security.credential.provider.path=<path> -Dfs.s3a.server-side-encryption-algorithm=####--options-file <path> --query "select cols from DB.Table where UPDATETIME between to_date('2015-09-11 00:00:00','yyyy/mm/dd hh24:mi:ss') and to_date('2018-05-24 04:28:16','yyyy/mm/dd hh24:mi:ss') and \$CONDITIONS" --hive-delims-replacement ' ' --direct --m 1 --split-by ID --target-dir <S3 path> --target-dir <hdfs path>
LUZO
  • 1,019
  • 4
  • 19
  • 42

1 Answers1

0

Sqoop doesn't support multiple targets. You can try a two step approach:

  1. Sqoop import from DB to HDFS
  2. distcp from HDFS to Amazon S3
Jagrut Sharma
  • 4,574
  • 3
  • 14
  • 19
  • tried but facing exception: Unable to load credentials from Amazon EC2 metadata service for distcp – LUZO May 25 '18 at 12:36