2

Is there a way to run a scalding job that needs class-path without using libjars and writing each jar explicitly comma separated.

I would like to put all my jars in a lib and than just write -libjars=./lib/* and not all the jars.

Is there a classic way ??

using scalding version scalding-core_2.10 0.8.5

hadoop version 2.0.0-cdh4.5.0

P.S I am using maven in order to package the jar and the lib

Ehud Lev
  • 2,461
  • 26
  • 38

1 Answers1

0

You should create a 'fat jar' containing both the dependencies you otherwise would add with libjars, as well as the main class. A concrete example for sbt project with Scalding and dependencies, in one jar, can be viewed here. In Maven, you should add the 'assembly plugin' and package using mvn assembly:first (if I recall). Expect that neither of the methods above would be easy since you're gonna have dependencies hell if you are going to use other jars that are already present in the cloudera jars (with different versions, for example). I wrote about something similar few years ago and you can try it out, although it might already be obsolete.

Amit
  • 1
  • 1
  • Thanks Amit, I am familiar with the fat jar, but I would like to use a slim jar cause we use git to deploy jars and it is easier to deploy several jars than 1 fat jar (assuming that most of the jars are not changing). As far as I looked there is no classic way to do that. – Ehud Lev Jul 27 '14 at 12:09