5

Simple case for me: I'm using 2 very popular but conflicting scala libraries: Spark SQL & JSON4s. Each depends on different versions of the Jackson XML parser.

Since nobody except my own program is using JSON4s, a simple solution is to migrate the packages org.json4s.jackson and com.fasterxml.jackson to new locations.

However, the maven-shade plugin can only do this in the packaging process, since all tests and IDE runs happen before that. This still cause all tests to fail, regardless of whether to be run in mvn-test or scala test in the IDE.

Is there a plugin that can collectively manage package relocation policies for all three cases: JAR-packaging/mvn-test/IDE-run? I've been searching the Internet all day and can't find an answer.

Lars Gendner
  • 1,816
  • 2
  • 14
  • 24
tribbloid
  • 4,026
  • 14
  • 64
  • 103
  • What are the different versions of Jackson? You could override one of the dependencies in your own pom.xml so that only one instance exists. – Tassos Bassoukos Sep 05 '15 at 22:45
  • You won't believe this: 3.2.10(Spark) vs 3.2.11(My code). If Spark jackson is overriden to 3.2.11 it has classNotFound error, if I change my jackson to 3.2.10 some of my tests and feature breaks. So I'm in a dilemma situation now – tribbloid Sep 05 '15 at 23:01
  • I'm only half-joking, but use OSGi to load both versions... – Tassos Bassoukos Sep 05 '15 at 23:12
  • Otherwise, can't you create relocated versions of the libraries you need in your own group ID? Clone, move/rename, mvn install and directly reuse? – Tassos Bassoukos Sep 05 '15 at 23:16
  • Would an older version of JSON4s using the same jackson as Spark be an option? – Thorbjørn Ravn Andersen Sep 05 '15 at 23:26
  • That's a lot of questions :) I don't know anything about OSGI so probably I'll tackle with it later. P4: I don't know if this can be done out of the box, do you have a maven plugin config in mind (anything that doesn't involve copying files to .m2 folder)? P5: No, I have to deserialize generic class that doesn't have "string" type parameter, in Json4s 3.2.10 this feature hasn't been added yet (and for unknown reason 3.2.11 is not backward compatible) – tribbloid Sep 05 '15 at 23:32
  • What exactly does " a simple solution is to migrate the package org.json4s.jackson and com.fasterxml.jackson to a new location." mean? – Thorbjørn Ravn Andersen Sep 05 '15 at 23:38
  • P7: this is a feature in maven-shade plugin when packaging into a fat jar: basically, it means moving all class files in json4s 3.2.11 into a new package that has a different name, and modify my java bytecode (or class loader, can't remember which one exactly) to use them. Since Spark's bytecode won't be changed, it can still safely use 3.2.10 without being broken. All this process happens automatically. – tribbloid Sep 05 '15 at 23:51
  • If you _must_ have the two in the same classloader (which you probably do) then I would create a new maven artifact in your current source tree where the maven shade plugin run on json4s as you describe, but that only, run "mvn clean install"and then rewrite your code to use the revised classnames of json4s. – Thorbjørn Ravn Andersen Sep 06 '15 at 06:59
  • That won't be helpful, the maven-shade plugin only works on packaging of fat jar, it doesn't rewrite any thing in the common jars that is released to the maven local repository (using 'mvn clean install'). – tribbloid Sep 06 '15 at 16:45
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/88951/discussion-between-tribbloid-and-thorbjorn-ravn-andersen). – tribbloid Sep 06 '15 at 20:50
  • Then use something else which can. – Thorbjørn Ravn Andersen Sep 06 '15 at 21:09
  • That's exactly my question, what is the thing that can? – tribbloid Sep 07 '15 at 15:26

1 Answers1

4

We solved the issue by recompiling (and patching) to align the libraries. Anything shade related is going to favor one version over the other, which means that what you might fix in one library you might break in another.

And for those comments from the OSGi camp, yes, that's what OSGi was meant to fix, but it doesn't work out so well in a Spark context :)

Have you considered downgrading one of the two versions to align resources? Often one of the libraries manages to release first, leaving the second one behind by just a bit... Sometimes the answer can be found in running a consistent yet older version that aligns with the "last to be released" project.

Edwin Buck
  • 69,361
  • 7
  • 100
  • 138
  • That sounds much harder than I thought: I used to believe that shaded artifacts are just got packages renamed and can co-exist with the unshaded one (as long as one of them is used by myself exclusively and not by other library). I have downgraded for other cases before, but in this case json4s-jackson 3.11 provide some important functionality that is almost impossible to be implemented on older version – tribbloid Sep 14 '15 at 02:43
  • @tribbloid You are not going to like this, but the shaded option is primarily for those who find it difficult to maintain a classpath. It literally unpacks the jars (as they are zip files) and copies them into the target jar (again zip files), so shading when you have to different versions of the same jar typically leaves you with one working version of that jar (the last one) and references to the items that are aware of the old jar sometimes breaks, depending on if the dependent classes happen to function identically to their replacements. – Edwin Buck Sep 14 '15 at 15:20
  • 1
    If possible, and if you don't want to take on the need to compile it all, I'd try to find a date where all the libraries align and then upgrade / downgrade components to get your desired release. Sometimes that means only patching and compiling a few items, instead of this huge ball of stuff. – Edwin Buck Sep 14 '15 at 15:22