1

I am following this guide on Github and I am not able run the example mapreduced job mentioned in Step 5.

I am aware that this file no longer exists:

/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar

And I am aware that the same file can now be found here:

/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u6.jar

So I form my call as below:

curl -v -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/$user/jobs" -d '{"jar":"/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u6.jar","class_name":"WordCount","lib_jars":"/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u6.jar","input":"testdir","output":"testoutput"}' -H "Content-Type: application/json" -H "X-Auth-Token: $TOKEN"

The input directory exists in my hdfs user space and there is a file called testdata.txt inside it. The testoutput folder does not exist in my hdfs user space since I know it creates problems.

When I execute this curl command, the error I get is {"success":"false","error":1} which is not very descriptive. Is there something I am missing here?

McMutton
  • 905
  • 13
  • 25

1 Answers1

1

This has been just tested with my user frb and a valid token for that user:

$ curl -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/frb/jobs" -d '{"jar":"/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar","class_name":"wordcount","lib_jars":"/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar","input":"testdir","output":"outputtest"}' -H "Content-Type: application/json" -H "X-Auth-Token: xxxxxxxxxxxxxxxxxxx"
{"success":"true","job_id": "job_1460639183882_0011"}

Please observe the fat jar with the MapReduce examples in the "new" cluster (computing.cosmos.lab.fiware.org) is at /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar, as detailed in the documentation. /usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u6.jar was the fat jar in the "old" cluster (cosmos.lab.fiware.org).

EDIT 1

Finally, the user had no account in the "new" pair of clusters of Cosmos in FIWARE LAB (storage.cosmos.lab.fiware.org and computing.cosmos.lab.fiware.org), where Tidoop runs, but in another "old" cluster (cosmos.lab.fiwre.org). Thus, the issue was fixed by simply provisioning an account in the "new" ones.

frb
  • 3,738
  • 2
  • 21
  • 51
  • Thank you for your reply. Even though I changed the path to the jar files on my curl command, I get the same response `{"success":"false","error":1}`. The only difference between our curl commands are the user name and the token. The token is valid since I can list, create, delete files and folders. Might this be related with user rights or, a long shot but, my user name? – McMutton May 19 '16 at 15:47
  • Wait a moment... you don't have an account in the new cluster! :) I've seen you have an account in the old one (`cosmos.lab.fiware.org`), but not in the new one (in fact, it is a pair of clusters, `storage.cosmos.lab.fiware.org` and `computing.cosmos.lab.fiware.org`). Please read the section 2 of the QSG for the new cluster (the one you linked) in order to get an account in the new cluster. – frb May 23 '16 at 10:41
  • Btw, Tidoop is only avaliable in the new pair of clusters. – frb May 23 '16 at 10:42
  • 1
    Thank you very much! I can now use the API. An update for the ones who are dealing with the same issue:My problem was not having a user account in the new cluster (storage.cosmos... and computing.cosmos...) I sent an e-mail to the address provided in the section 2 of the QSG about the problem and, once my user was provisioned to the new cluster, I was able to use the API for running Hadoop jobs. – McMutton May 23 '16 at 16:03