AWS s3 sync is slow. Also, AWS s3 sync silently fails. s3 mv deletes target files on failure and cleans up. However, it does not attempt to copy(needs error handling). Whats my best option to move files from s3 to EMR? - s3 cmd? - s3 mv? Any other suggestion?
Asked
Active
Viewed 224 times
1 Answers
0
As the maintainer of the project, I would recommend you to use s3cmd as it is a core idea to not fail silently or corrupt your files. I would also recommend you to use the most recent version when possible, as a lot of great improvements arrived recently and more are coming.
A tip to conclude, that should be true for whatever tool you finally plan on using: don't trust the tool blindly, try to understand the help and the meaning of commands, and test carefully the operations you want to do before putting in production, to be sure that you have the result that you expected.
Florent V
- 191
- 1
- 7