Optional dependencies of external library added via Grab not available

Question

It is not possible to use external third party libaries with optional dependencies properly in shared libraries for Jenkins.

I have a shared library which uses Commons Configurations 2 to read varios configuration files, mostly written as YAML documents.

Commons configurations uses SnakeYAML to read YAML documents and the dependency to SnakeYAML is defined as optional as follows:

 <dependency>
      <groupId>org.yaml</groupId>
      <artifactId>snakeyaml</artifactId>
      <version>1.26</version>
      <optional>true</optional>
 </dependency>

According to the documentation of Maven how optional dependencies work an optional dependency is not added by default to the classpath. If a person wants to the part of the library which depends on the optional dependency, it must add this dependency to his own POM.

As I was going to use Commons Configuration 2 in conjunction with SnakeYAML, I defined the following variable in vars/readConfig.groovy as follows:

@Grapes([
  @Grab(group = "org.apache.commons", module = "commons-configuration2", version = "2.7"),
  @Grab(group = "org.yaml", module = "snakeyaml", version = "1.26")
])
import org.apache.commons.configuration2.BaseConfiguration
import org.apache.commons.configuration2.Configuration
import org.apache.commons.configuration2.YAMLConfiguration

def call() {
    Configuration config = new BaseConfiguration();
    YAMLConfiguration yamlConfiguration = new YAMLConfiguration();  
}

Calling readConfig() from a shared Library results in a java.lang.ClassNotFoundException with the following message

java.lang.ClassNotFoundException: org.yaml.snakeyaml.DumperOptions
    at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1387)
    at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1342)
    at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1089)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetPublicMethods(Class.java:2902)
    at java.lang.Class.getMethods(Class.java:1615)
    at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1336)
    at java.beans.Introspector.getTargetMethodInfo(Introspector.java:1197)
    at java.beans.Introspector.getBeanInfo(Introspector.java:426)
    at java.beans.Introspector.getBeanInfo(Introspector.java:173)
    at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:3313)
    at java.security.AccessController.doPrivileged(Native Method)
    at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3311)
    at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3288)
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:260)
    at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:302)
    ...

I also checked the directory ~/.groovy/grapes for the presence of all needed jars and they are there.

jenkins@cb765137c926:~/.groovy$ find . -name "*.jar"
./grapes/commons-logging/commons-logging/jars/commons-logging-1.2.jar
./grapes/org.apache.commons/commons-configuration2/jars/commons-configuration2-2.7.jar
./grapes/org.apache.commons/commons-lang3/jars/commons-lang3-3.9.jar
./grapes/org.apache.commons/commons-text/jars/commons-text-1.8.jar
./grapes/org.yaml/snakeyaml/jars/snakeyaml-1.26.jar

To do a cross-check I wrote the following Groovy script and was able to execute it on my computer successfully.

@Grapes([
  @Grab(group = 'org.apache.commons', module = 'commons-configuration2', version = '2.7'),
  @Grab(group = 'org.yaml', module = 'snakeyaml', version = '1.26'),
  @GrabConfig(systemClassLoader = true)
])

import org.apache.commons.configuration2.*

println("Start")

YAMLConfiguration y = new YAMLConfiguration()

println y

So, I am not able to guess the reason for this problem, as I am not so familiar with the internals of Jenkins. But it would be great to know if there is a way to get it working as intended.

try to import and create `org.yaml.snakeyaml.DumperOptions`. — daggett, May 18 '20 at 22:31
Hi @daggett, it is possible to create `org.yaml.snakeyaml.DumperOptions` and it is also possible to create an instance of `org.apache.commons.text.similarity.JaroWinklerDistance`. This class comes from Commons Text, which is a non-optional depenendency of Commons Configurations 2. — Oliver, May 19 '20 at 06:45
Created an issue for this problem in Jenkins bugtracker: https://issues.jenkins-ci.org/browse/JENKINS-62387 — Oliver, May 21 '20 at 14:34
that could be a problem of `commons-configuration2` class loader — daggett, May 21 '20 at 17:28

score 1 · Answer 1 · answered May 27 '20 at 15:44

@Grab will not work in the sandbox, so this will only work in a shared library defined in the global configuration. Even when it does, you are quite likely to run into class loading issues, because of the way Jenkins creates an environment for your script to run in.

Cloudbees recommends against using @Grab in shared libraries, for this reason, but also for performance reasons. What this code will do, when you get it to work, is download those dependencies, in blocking calls, over the internet, at the start of every single build on that master. You can very quickly run out of executors this way if the network is slow.

Best practice is to install the Pipeline Utility Steps Plugin, which provides the readYaml method. It's just based on SnakeYaml sources, it will work the same.

Hi @Mzzl, thank you for your answer. I run several tests and the library will be downloaded only once. After the inital download it is keept in the `.groovy/grapes` directory and every subsequent call uses the downloaded library artifact. — Oliver, May 27 '20 at 21:53

score 1 · Accepted Answer · edited May 27 '20 at 22:14

1

Grab is built upon Ivy. What we do in our Ivy config to get optional dependencies is to add optional to the element conf of the annotation.

@Grab(group='org.apache.commons', module='commons-configuration2', 
      version='2.7', conf='default,optional')

edited May 27 '20 at 22:14

Oliver

3,815
8
35
63

answered May 27 '20 at 17:17

emilles

1,104
7
14

Optional dependencies of external library added via Grab not available

2 Answers2