Best way to copy Hive data from HDFS to another machine?

Asked Jun 23 '14 at 18:47

Active Jun 23 '14 at 18:47

Viewed 455 times

Source: /mnt/fuse/project/hive (This is hive data on HDFS mounted using fuse)

Destination: host:/fs/dir (/fs/dir is again a network mounted storage and I do not have direct access to "host")

I did rsync -avz -P -e ssh /mnt/fuse/project/hive user@host:/fs/dir and this is very slow. I aborted the process and tried to kill rsync but could not do so, tried both kill -15 and kill -9.

Wondering if there is a better way to copy the data?

asked Jun 23 '14 at 18:47

Sambit Tripathy

did you try distcp? The command line could be something like `hadoop distcp hdfs:///path/to/hivedata fs:///user@host:/fs/dir` – zhutoulala Jun 23 '14 at 23:10
I think distcp is used to copy data in between clusters. – Sambit Tripathy Jun 24 '14 at 00:11
oh, it could be used with multiple schemes, i.e. hdfs, http, ftp, file..., i used it for cp files from HDFS to local file system with file:///, but i didn't try your case, that's the reason i suggested you to have a try – zhutoulala Jun 24 '14 at 00:27
I have tried it and it throws me an exception "fs: scheme not found" – Sambit Tripathy Jun 24 '14 at 02:45

Best way to copy Hive data from HDFS to another machine?

0 Answers0