Maven Shading Multi-Module Aggregation and Inheritance

Question

Context

I have an IntelliJ project with multiple modules. I am using the maven-shade-plugin to jar with dependencies whilst reducing jar size. The project structure is as follows;

top-level parent (pom) (aggregates api and world-teleport)
- api (inherits parent)
- world-teleport (inherits parent, api)

Pom(s)

Parent

   <groupId>com.jsonmack</groupId>
    <artifactId>mcplugins</artifactId>
    <packaging>pom</packaging>
    <version>1.10-SNAPSHOT</version>

    <repositories>
        <repository>
            <id>spigot-repo</id>
            <url>https://hub.spigotmc.org/nexus/content/repositories/snapshots/</url>
        </repository>
    </repositories>

    <modules>
        <module>api</module>
        <module>world-teleport</module>
    </modules>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
                <version>3.8.1</version>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <minimizeJar>true</minimizeJar>
                            <artifactSet>
                                <excludes>
                                    <exclude>junit:junit</exclude>
                                </excludes>
                            </artifactSet>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.github.classgraph</groupId>
                <artifactId>classgraph</artifactId>
                <version>4.8.68</version>
            </dependency>

            <dependency>
                <groupId>org.spigotmc</groupId>
                <artifactId>spigot-api</artifactId>
                <version>1.15.2-R0.1-SNAPSHOT</version>
            </dependency>

            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>4.13</version>
                <scope>test</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

api

    <parent>
        <artifactId>mcplugins</artifactId>
        <groupId>com.jsonmack</groupId>
        <version>1.10-SNAPSHOT</version>
    </parent>

    <modelVersion>4.0.0</modelVersion>
    <artifactId>api</artifactId>
    <packaging>jar</packaging>

    <dependencies>
        <dependency>
            <groupId>io.github.classgraph</groupId>
            <artifactId>classgraph</artifactId>
        </dependency>

        <dependency>
            <groupId>org.spigotmc</groupId>
            <artifactId>spigot-api</artifactId>
        </dependency>
    </dependencies>

world-teleport

   <parent>
        <groupId>com.jsonmack</groupId>
        <artifactId>mcplugins</artifactId>
        <version>1.10-SNAPSHOT</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>com.jsonmack</groupId>
            <artifactId>api</artifactId>
            <version>1.10-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>io.github.classgraph</groupId>
            <artifactId>classgraph</artifactId>
        </dependency>

        <dependency>
            <groupId>org.spigotmc</groupId>
            <artifactId>spigot-api</artifactId>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
        </dependency>
    </dependencies>

The problem

Everytime I perform a clean and a new install or package it is indicated to me that there are duplicate dependencies.

The error/warning(s)

[INFO] 
[INFO] --------------------< com.jsonmack:world_teleport >---------------------
[INFO] Building world_teleport 1.8-SNAPSHOT                               [7/7]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ world_teleport ---
[INFO] Deleting /Users/Business/Documents/workspace/mcplugins/world_teleport/target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ world_teleport ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ world_teleport ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 19 source files to /Users/Business/Documents/workspace/mcplugins/world_teleport/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ world_teleport ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/Business/Documents/workspace/mcplugins/world_teleport/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ world_teleport ---
[INFO] Changes detected - recompiling the module!
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ world_teleport ---
[INFO] 
[INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ world_teleport ---
[INFO] Building jar: /Users/Business/Documents/workspace/mcplugins/world_teleport/target/world_teleport-1.8-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-shade-plugin:3.2.2:shade (default) @ world_teleport ---
[INFO] Including com.jsonmack:api:jar:1.10-SNAPSHOT in the shaded jar.
[INFO] Including io.github.classgraph:classgraph:jar:4.8.68 in the shaded jar.
[INFO] Including org.spigotmc:spigot-api:jar:1.15.2-R0.1-SNAPSHOT in the shaded jar.
[INFO] Including commons-lang:commons-lang:jar:2.6 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:21.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.8.0 in the shaded jar.
[INFO] Including net.md-5:bungeecord-chat:jar:1.15-SNAPSHOT in the shaded jar.
[INFO] Including org.yaml:snakeyaml:jar:1.25 in the shaded jar.
[INFO] Minimizing jar com.jsonmack:world_teleport:jar:1.8-SNAPSHOT
[WARNING] /Users/Business/Documents/workspace/mcplugins/world_teleport/target/classes (Is a directory)
[WARNING] api-1.10-SNAPSHOT.jar, snakeyaml-1.25.jar define 207 overlapping classes and resources: 
[WARNING]   - META-INF/maven/org.yaml/snakeyaml/pom.properties
[WARNING]   - META-INF/maven/org.yaml/snakeyaml/pom.xml
[WARNING]   - org.yaml.snakeyaml.DumperOptions
[WARNING]   - org.yaml.snakeyaml.DumperOptions$FlowStyle
[WARNING]   - org.yaml.snakeyaml.DumperOptions$LineBreak
[WARNING]   - org.yaml.snakeyaml.DumperOptions$NonPrintableStyle
[WARNING]   - org.yaml.snakeyaml.DumperOptions$ScalarStyle
[WARNING]   - org.yaml.snakeyaml.DumperOptions$Version
[WARNING]   - org.yaml.snakeyaml.LoaderOptions
[WARNING]   - org.yaml.snakeyaml.TypeDescription
[WARNING]   - 197 more...
[WARNING] api-1.10-SNAPSHOT.jar, guava-21.0.jar define 931 overlapping classes and resources: 
[WARNING]   - META-INF/maven/com.google.guava/guava/pom.properties
[WARNING]   - META-INF/maven/com.google.guava/guava/pom.xml
[WARNING]   - com.google.common.annotations.Beta
[WARNING]   - com.google.common.annotations.GwtCompatible
[WARNING]   - com.google.common.annotations.GwtIncompatible
[WARNING]   - com.google.common.annotations.VisibleForTesting
[WARNING]   - com.google.common.base.Absent
[WARNING]   - com.google.common.base.AbstractIterator
[WARNING]   - com.google.common.base.AbstractIterator$1
[WARNING]   - com.google.common.base.AbstractIterator$State
[WARNING]   - 921 more...
[WARNING] api-1.10-SNAPSHOT.jar, gson-2.8.0.jar define 161 overlapping classes and resources: 
[WARNING]   - META-INF/maven/com.google.code.gson/gson/pom.properties
[WARNING]   - META-INF/maven/com.google.code.gson/gson/pom.xml
[WARNING]   - com.google.gson.ExclusionStrategy
[WARNING]   - com.google.gson.FieldAttributes
[WARNING]   - com.google.gson.FieldNamingPolicy
[WARNING]   - com.google.gson.FieldNamingPolicy$1
[WARNING]   - com.google.gson.FieldNamingPolicy$2
[WARNING]   - com.google.gson.FieldNamingPolicy$3
[WARNING]   - com.google.gson.FieldNamingPolicy$4
[WARNING]   - com.google.gson.FieldNamingPolicy$5
[WARNING]   - 151 more...
[WARNING] api-1.10-SNAPSHOT.jar, bungeecord-chat-1.15-SNAPSHOT.jar define 17 overlapping classes and resources: 
[WARNING]   - META-INF/maven/net.md-5/bungeecord-chat/pom.properties
[WARNING]   - META-INF/maven/net.md-5/bungeecord-chat/pom.xml
[WARNING]   - mojang-translations/en_US.properties
[WARNING]   - mojang-translations/en_us.json
[WARNING]   - net.md_5.bungee.api.ChatColor
[WARNING]   - net.md_5.bungee.api.ChatMessageType
[WARNING]   - net.md_5.bungee.api.chat.BaseComponent
[WARNING]   - net.md_5.bungee.api.chat.ClickEvent
[WARNING]   - net.md_5.bungee.api.chat.ClickEvent$Action
[WARNING]   - net.md_5.bungee.api.chat.ComponentBuilder
[WARNING]   - 7 more...
[WARNING] api-1.10-SNAPSHOT.jar, commons-lang-2.6.jar define 63 overlapping classes and resources: 
[WARNING]   - META-INF/LICENSE.txt
[WARNING]   - META-INF/NOTICE.txt
[WARNING]   - META-INF/maven/commons-lang/commons-lang/pom.properties
[WARNING]   - META-INF/maven/commons-lang/commons-lang/pom.xml
[WARNING]   - org.apache.commons.lang.ArrayUtils
[WARNING]   - org.apache.commons.lang.BooleanUtils
[WARNING]   - org.apache.commons.lang.CharRange
[WARNING]   - org.apache.commons.lang.CharRange$1
[WARNING]   - org.apache.commons.lang.CharRange$CharacterIterator
[WARNING]   - org.apache.commons.lang.CharSet
[WARNING]   - 53 more...
[WARNING] api-1.10-SNAPSHOT.jar, bungeecord-chat-1.15-SNAPSHOT.jar, classgraph-4.8.68.jar, commons-lang-2.6.jar, gson-2.8.0.jar, guava-21.0.jar, snakeyaml-1.25.jar, spigot-api-1.15.2-R0.1-SNAPSHOT.jar, world_teleport-1.8-SNAPSHOT.jar define 1 overlapping resources: 
[WARNING]   - META-INF/MANIFEST.MF
[WARNING] api-1.10-SNAPSHOT.jar, spigot-api-1.15.2-R0.1-SNAPSHOT.jar define 710 overlapping classes and resources: 
[WARNING]   - META-INF/maven/org.spigotmc/spigot-api/pom.properties
[WARNING]   - META-INF/maven/org.spigotmc/spigot-api/pom.xml
[WARNING]   - org.bukkit.Art
[WARNING]   - org.bukkit.Axis
[WARNING]   - org.bukkit.BanEntry
[WARNING]   - org.bukkit.BanList
[WARNING]   - org.bukkit.BanList$Type
[WARNING]   - org.bukkit.BlockChangeDelegate
[WARNING]   - org.bukkit.Bukkit
[WARNING]   - org.bukkit.ChatColor
[WARNING]   - 700 more...
[WARNING] api-1.10-SNAPSHOT.jar, classgraph-4.8.68.jar define 228 overlapping classes and resources: 
[WARNING]   - LICENSE-ClassGraph.txt
[WARNING]   - META-INF.versions.9.module-info
[WARNING]   - META-INF/maven/io.github.classgraph/classgraph/pom.properties
[WARNING]   - META-INF/maven/io.github.classgraph/classgraph/pom.xml
[WARNING]   - io.github.classgraph.AnnotationClassRef
[WARNING]   - io.github.classgraph.AnnotationEnumValue
[WARNING]   - io.github.classgraph.AnnotationInfo
[WARNING]   - io.github.classgraph.AnnotationInfo$AnnotationInvocationHandler
[WARNING]   - io.github.classgraph.AnnotationInfoList
[WARNING]   - io.github.classgraph.AnnotationInfoList$AnnotationInfoFilter
[WARNING]   - 218 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
[INFO] Minimized 6106 -> 4706 (77%)

Question

What is causing this warning? What if anything is wrong with my maven pom structure?

Duplicate Question

If this is a duplicate question please feel free to provide any resources, they would be greatly appreciated.

score 1 · Answer 1 · answered Apr 14 '20 at 13:42

I think the problem is that you have the io.github.classgraph and the org.spigotmc dependency added in the api and the module-1 as well.

So what happens:

The api module will generate the jar artifact with io.github.classgraphand the org.spigotmc dependencies included in it.
The module-1 modul which depends on the api module will import the jar file.
Now the module-1 module has the io.github.classgraphand the org.spigotmc dependency from the jar, and they are declared again in the module-1 module's pom.xml file.

So the solution will be to remove the io.github.classgraph and the org.spigotmc dependency from your module-1 module pom.xml:

<parent>
    <groupId>com.jsonmack</groupId>
    <artifactId>mcplugins</artifactId>
    <version>1.10-SNAPSHOT</version>
</parent>

<dependencies>
    <dependency>
        <groupId>com.jsonmack</groupId>
        <artifactId>api</artifactId>
        <version>1.10-SNAPSHOT</version>
    </dependency>

    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
    </dependency>
</dependencies>

Unfortunately this did not resolve the problem. This was a very good idea however. I have updated the original post with more POM information to help with debugging. +1 for taking the time to comment. — Jason, Apr 14 '20 at 14:23
The amount of warnings is the same. The warning about the API dependency having duplicate dependencies is unfortunately still very apparent. — Jason, Apr 14 '20 at 14:41

score 1 · Answer 2 · answered Apr 14 '20 at 13:49

Well here I would like to appreciate your proactive was of noticing that as many developers keep on adding decencies and do not bother about such warnings.

Answer my question, as you have three different modules. Will those modules be deployed differently (independently) or together in one war or jar or package?

If your answer is yes here is nothing to worry.

If your answer is no then I see io.github dependency as redundant just get rid of it.

Delete the redundant dependencies from child module or referenced module.

Sorry if there are typos, answered in mobile but will do the job for you.

api module is package seperately. world-teleport is package with api as a dependency. +1 for trying to help but unfortunately does not resolve the problem. — Jason, Apr 14 '20 at 14:26

score 1 · Answer 3 · answered Apr 14 '20 at 14:44

1

An other idea: Try removing io.github.classgraph and the org.spigotmc dependencies from the parent pom as well.

Parent

<groupId>com.jsonmack</groupId>
<artifactId>mcplugins</artifactId>
<packaging>pom</packaging>
<version>1.10-SNAPSHOT</version>

<repositories>
    <repository>
        <id>spigot-repo</id>
        <url>https://hub.spigotmc.org/nexus/content/repositories/snapshots/</url>
    </repository>
</repositories>

<modules>
    <module>api</module>
    <module>world-teleport</module>
</modules>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
            <version>3.8.1</version>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.2</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <minimizeJar>true</minimizeJar>
                        <artifactSet>
                            <excludes>
                                <exclude>junit:junit</exclude>
                            </excludes>
                        </artifactSet>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

world-teleport

<parent>
    <groupId>com.jsonmack</groupId>
    <artifactId>mcplugins</artifactId>
    <version>1.10-SNAPSHOT</version>
</parent>

<dependencies>
    <dependency>
        <groupId>com.jsonmack</groupId>
        <artifactId>api</artifactId>
        <version>1.10-SNAPSHOT</version>
    </dependency>

    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
    </dependency>
</dependencies>

answered Apr 14 '20 at 14:44

xBeke

101
5

Unfortunately I would have to update each individual pom when a dependency updates, which is why I'm using inheritance so each child module inherits the version from the parent. This doesn't resolve the problem because API still has the defined dependencies and so does the module that API uses. I might try to exclude dependencies from API when packaging since they will always be included in a module that uses API. I should be able to do this using compile instead of provided as scope of depedency. I'm on mobile but I'll try shortly. – Jason Apr 14 '20 at 14:48
If you delete the dependency from the parent and the world-teleport then you will still be able to update the dependency from the api. 'cause if you update the dependency from there, then next tine the updated dependency will be imported into world-teleport module. – xBeke Apr 14 '20 at 14:53
There is no need to declare it in the parent. If you would not import the api, into world-teleport then yes you would need to declare it the parent. I think. But i could be wrong :D – xBeke Apr 14 '20 at 14:55
Not all modules make use of the api, so it's necessary. That might not have been included in the original post so I apologize. I.e module-2 might make use of parent, but not api. – Jason Apr 14 '20 at 15:03
Last idea: What if the problem is that api and world-teleport generates an artifact and the shade-plugin detects that there is two jar with some dependencies that are identical. – xBeke Apr 14 '20 at 15:53
So actually there is no problem :D – xBeke Apr 14 '20 at 15:53
I provided an answer above that resolved the problem. Thanks for all the help. I can provide the bounty to you in 20ish hours for the help. If you don't get it in 20 link me on here to remind me :) Thanks. – Jason Apr 14 '20 at 15:56
Daaamn i did not thought about that solution. Nice :) And what bounty do you talk about? – xBeke Apr 14 '20 at 20:50
I gave 50 points to have this resolved, but I can choose an answer in 24 hours from the time I made a bounty and give it to a person. I can't get the 50 points back so might as well give it to you for helping me. I dont think I would have come to this solution at this time without our discussion to get the juices flowing. – Jason Apr 14 '20 at 21:35
Is this possible that i did not get that? – xBeke Apr 16 '20 at 09:30
1

Yap i got it! :) Thank you :) – xBeke Apr 16 '20 at 10:16

score 1 · Answer 4 · answered Apr 14 '20 at 15:37

Thank you for the help everyone. I have found the solution which is that the API module should inherit the dependencies such as spigot and classgraph from parent but the scope of those dependencies should only be provided. This will allow the dependencies to compile but be excluded from packaging when shading. This is possible because API will not be used outside of these modules, so the API will always have the dependencies required through the other modules.

api-module.pom

    <dependencies>
        <dependency>
            <groupId>io.github.classgraph</groupId>
            <artifactId>classgraph</artifactId>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.spigotmc</groupId>
            <artifactId>spigot-api</artifactId>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>