5

I have a large multimodule maven project that has a large number of dependencies. I would like to generate a complete, duplicate-filtered list of third-party dependencies (that is, all dependencies not using the group id of the project) this project has.

I have tried using mvn dependency:list -DexcludeGroupIds=org.example.projectx for this purpose, but it seems unable to aggregate the output into a single list. When I run the command from the project root command line, I get output as follows:

[...]
[INFO] ------------------------------------------------------------------------
[INFO] Building ProjectX: ModuleA - Datatypes 4.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:list (default-cli) @ projectx-moda-datatypes ---
[INFO] 
[INFO] The following files have been resolved:
[INFO]    org.slf4j:slf4j-api:jar:1.7.10:compile
[INFO]    org.hamcrest:hamcrest-core:jar:1.3:test
[INFO]    junit:junit:jar:4.12:test
[INFO]    com.google.guava:guava:jar:18.0:compile
[INFO] 
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building ProjectX: ModuleB - Binary 4.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:list (default-cli) @ projectx-modb-binary ---
[INFO] 
[INFO] The following files have been resolved:
[INFO]    org.slf4j:slf4j-api:jar:1.7.10:compile
[INFO]    com.google.guava:guava:jar:18.0:compile

..etc, for every single submodule. Not only is this not a single list (but a separate list for each submodule), but as you can see it contains duplicates. Moreover, the actual output I'm interested in is buried in a torrent of other Maven output (download messages, etc).

The -DoutputFile=<file.txt> option does not offer a solution either. The result of running mvn dependency:list -DoutputFile=deps.txt from my project root is not a single file listing all dependencies, but multiple separate files, one in each submodule-directory.

I can of course redirect the maven console output to a file (mvn [options] > output.txt) and try some clever regexing in vi to filter it down to the list I want. However, I was hoping there was a way to get what I need using just Maven, either using the dependency plugin or some other reporting plugin that I'm not aware of.

A_Di-Matteo
  • 26,902
  • 7
  • 94
  • 128
Jeen Broekstra
  • 21,642
  • 4
  • 51
  • 73

3 Answers3

4

You could create an additional module (or an external maven project) having pom packaging (that is, empty) and having as dependencies all of the modules/projects you are interested in. Then, run the site life cycle on it (that is, simply mvn site). As part of the generated site under target\site, you will have:

  • Project Dependencies page, providing you direct dependencies (not helpful, it will be the list of dependencies you typed) AND transitive dependencies (what you are looking for) all merged in one list with GAV, type, licence. Moreover, you will also have the full dependency:tree merged.
  • Dependency Convergence page, providing really useful information about convergence of the same transitive dependency and its different versions across other dependencies, if any

Alternatively, you could also use this ad-hoc module/project to run other plugins on it, like the dependency:list you mentioned. This time you will get the full merged list you were looking for, although not alphabetically ordered and still difficult to read/re-use since it will be part of the build output. The site reports is instead much easier to read and handle (i.e. copy and paste of the dependencies table to an excel file).

For reference, the Maven Project Info Reports Plugin will be used by the site phase. Official site, here.

A_Di-Matteo
  • 26,902
  • 7
  • 94
  • 128
  • Excellent! Fortunately, my project actually already has such a module, the BOM (Bill Of Materials). It's not useful for the dependency plugin since it only has a `dependencyManagement` section (so `dependency:list` just outputs an empty list), but running `mvn site` in there gives me almost exactly what I need. – Jeen Broekstra Nov 29 '15 at 22:46
  • Great, if that satisfies you, then better to accept the answer as final one in order not to leave unanswered the question for the future. Thanks. – A_Di-Matteo Nov 30 '15 at 10:50
4

I was trying out a similar exercise and could not find any configurations / goals to dependency plugin that can help in this regard. But fortunately, merging few commands would make it work well to provide a list of all unique and sorted list of all the dependencies.

mvn dependency:list | grep "\[INFO\][^:]*:[^:]*:[^:]*:[^:]*:.*" | cut -d] -f2- | sort | uniq

Being specific to this question, the below should work.

mvn dependency:list -DexcludeGroupIds=org.example.projectx | grep "\[INFO\][^:]*:[^:]*:[^:]*:[^:]*:.*" | cut -d] -f2- | sort | uniq
Pavan Kumar
  • 4,182
  • 1
  • 30
  • 45
  • 1
    In the first command uniq should be called after the sort. This worked like a charm otherwise. – Bill O'Neil May 25 '22 at 18:35
  • 1
    Not sure how it should make difference. Did not test it now, but made the change in answer as that even matches with the second command. – Pavan Kumar May 26 '22 at 12:21
  • 1
    I believe it's because the uniq only removes duplicate consecutive lines. When I ran this across a large multi module project it still had several duplicate lines when sort was called after uniq. – Bill O'Neil May 27 '22 at 13:04
0

I needed to extract the list of libraries/test libraries to feed them to the sonar scan application. We are not using the maven sonar-scanner plugin and I needed to have a simple way to avoid the developers to define the list of libraries manually. Also, I wanted to avoid to generally pass the whole maven cache to the scanner, as it would slow the scan down.

With the following commands, I can extract the list of dependencies and format as sorted, unique, comma separate list of full paths:

# This will generate the dependency list for each module and store it in separate files (#1)
mvn dependency:list -DoutputAbsoluteArtifactFilename=true -DoutputFile=dependencies.txt -Dsort=true --offline

# To print the test dependencies (#2)
{ find . -name dependencies.txt -exec grep ":test:" {} \;; } | sort | uniq | cut -d":" -f 6 | paste -d , -s

# To print the list of all dependencies except the test ones (#3)
{ find . -name dependencies.txt -exec grep -E ":.+:" {} \;; } | grep -v ":test:" | sort | uniq | cut -d":" -f 6 | paste -d , -s

Description:

  1. use the dependency plugin to generate all the dependencies and store them in files named dependencies.txt which will be stored in each module's folder. Also, it sorts the dependencies and prints the full path of the artifact (i.e. org.springframework.boot:spring-boot-autoconfigure:jar:2.5.6:compile:/usr/share/maven/ref/repository/org/springframework/boot/spring-boot-autoconfigure/2.5.6/spring-boot-autoconfigure-2.5.6.jar)

  2. find, in the current folder and subfolders, all the files named dependencies.txt and pass each found file to grep. Then filter only the lines containing ":test:", then sort all the lines (each line is a dependency), filter out duplicates, extract the 6th field (which correspond to the file name with full path), then paste it as a sequence of comma separated entries

  3. is similar to 1 but grep will filter all the lines with at least 2 ":" (some lines are descriptive and contain only 1), then the result is passed through grep again to filter out all the lines containing the ":test:" string, leaving only the runtime, compile etc dependencies.

Other info

  • the external curly brackets ("{ ... }") are used to create a sub shell
  • the ones together ("{}") are a placeholder for the find command and will be substituted with the path of each found file
  • the first ";" in "\;;" is to end the find command and the second is to end the command in the subshell
Fabio
  • 491
  • 3
  • 10