s3distcp fail with "mapreduce_shuffle does not exist"

Question

When I running command below,

s3-dist-cp --src s3://test/9.19 --dest hdfs:///user/hadoop/test

I got a error about auxService.

20/02/03 07:52:13 INFO mapreduce.Job: Task Id : attempt_1580716305878_0001_m_000000_2, Status : FAILED
Container launch failed for container_1580716305878_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

In many QnA, I found a solution like this link.

But there is no process for nodemanager.

[hadoop@ip-172-31-37-115 ~]$ initctl list | grep yarn
hadoop-yarn-timelineserver start/running, process 8149
hadoop-yarn-resourcemanager start/running, process 17331
hadoop-yarn-proxyserver start/running, process 8147

My EMR was created by quick menu with emr-5.28.0. Is there anyone knows about this problem?

Thanks!

I think it's `hadoop-yarn-resourcemanager` https://stackoverflow.com/a/43445699/4549682 But even after adding the mapreduce_shuffle aux service in the xml file, it didn't work for me. I wonder if this is some setting that has to be done upon spinning up the cluster. — wordsforthewise, Feb 20 '20 at 22:22
Also, I don't have the s3-dist-cp command available on my EMR, did you do anything special to get it? — wordsforthewise, Feb 20 '20 at 22:23

wordsforthewise · Answer 1 · 2020-02-21T00:33:37.053

I'm sure there's some way to update the configs, but what I did was create a cluster using the 'advanced' setup and chose these software packages:

Ganglia Hive Hue Mahout Pig Tez Spark Hadoop (8 in total)

Most of those, except spark, are installed with the default settings (the first radio button for software packages in quick setup). One of these software packages or something related to it is what causes s3-dist-cp to be installed, and I was able to use it with no problems with that setup.

s3distcp fail with "mapreduce_shuffle does not exist"

1 Answers1