2

I'm using maven shade plugin in order to create an uber jar while relocating all classes. That's because I'm getting an external jar and I don't want to have classpath collisions. So the idea is to create a new uber (relocated) jar and use it in my application. So the shade plugin takes all the classes and relocates them to the new package prefix. My issue is that a far as I understand, it also does that for dependencies which the classes they depend on, aren't in the scope [*].

Let's say I'm relocating all com to shade.com:

<executions>
  <execution>
    <id>rename-all</id>
    <phase>package</phase>
    <goals>
      <goal>shade</goal>
    </goals>
    <configuration>
      <shadedArtifactAttached>true</shadedArtifactAttached>
      <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
        <relocations>
          <relocation>
            <pattern>com</pattern>
            <shadedPattern>shade/com</shadedPattern>
          </relocation>
        </relocations>
        </configuration>
    </execution>
</executions>

So if one of my dependencies, A, has a <optional> dependency on B (which package com.B), the plugin would change the imports within A to point to shade.com.B. But com.B is optional, which means those classes won't be available in shade.com.B, because they didn't get into the relocation process. The actual classes available when using this jar would be the "normal" ones - com.B. And then I get class not found exceptions on shade.com.B classes when I try to use the shaded jar in my application.

Am I missing something in my understanding? Is there any solution to this?

[*] Some examples: I'm not sure yet about the exact cases where this happens. In my case, I depend on spark-sqldependency. I dig into 3 of the classes which I saw this issue (there are many more):

  1. org/apache/html/dom/HTMLIsIndexElementImpl imports org.w3c.dom.html.HTMLIsIndexElement which is in rt.jar - the import was relocated so now it points to shade.org.w3c.dom.html.HTMLIsIndexElement and can't be found.
  2. io/netty/handler/codec/marshalling/ChannelBufferByteOutput imports org.jboss.marshalling.ByteOutput. Hence the relocation change this to import shade.org.jboss.marshalling.ByteOutput. But as can be seen here, the import for jboss-marshalling is marked as <optional>true</optional>, so it's not in the jar and ByteOutput itself won't be relocated, hence won't be available.
  3. jasper-runtime is included here but excluded in the upper one, here. So the outcome is that the files in Hadoop-hdfs are getting relocation to their dependencies, even though these classes would never be available in the relocated path since they aren’t in the jar at all. e.g. org/apache/hadoop/hdfs/server/datanode/browseBlock_jsp now has a reference to “shade/org/apache/jasper/runtime/JspSourceDependent”.
yishaiz
  • 2,433
  • 4
  • 28
  • 49
  • have you checked that how? `My issue is that a far as I understand, it also does that for dependencies in scope.`? Apart from that usually using shaded jar's in applications is a really bad idea cause it produces issues in several way (you can't exclude etc.) and can not use the usual dependency mechanism which is really bad. My suggestion is: Only use shade to create executables but never for reusable artifacts... If your external jar is the problem fix that?.... otherwise it will cause more issues than you think / and like to have... – khmarbaise Nov 20 '20 at 16:38
  • I took the shaded jar and checked the compiled classes. I can see the imports. I've edited my question in order to provide some examples. Indeed, so far I didn't find an example with the `provided` scope - but I randomly chose 3 classes and checked why they fail, each one with different reason. – yishaiz Nov 23 '20 at 21:58
  • @khmarbaise I'm using the shaded jar as a pluggable jar for my application, the purpose is to bring it in and avoid possible class path collisions, hence the shading and uber jar. – yishaiz Nov 23 '20 at 21:59
  • You can of course do what you like but I strongly recommend not to do that...because it will produce error/problems in the future... – khmarbaise Nov 23 '20 at 22:22
  • Could you a little elaborate on which kind of problems? Assuming that I (successfully) shade the jar. – yishaiz Nov 24 '20 at 06:47

0 Answers0