Shibboleth IdP won't start with Jetty 9.3

Question

I've been following this guide to install Shibboleth Identity Provider 3.3.2 on Jetty 9.3.22 (Ubuntu Server 16.04 with Oracle JDK 8u151).

At https://my.server.ip/idp I get a 503 error (Service Unavailable).

Jetty Home: /opt/jetty/jetty-distribution (symlinked to jetty-distribution-9.3.22.v20171030 in the same directory)
Jetty Base: /opt/jetty/jetty-base
Shibboleth Installation: /opt/shibboleth-idp

I've created a jetty user and I'm using setuid to bind to ports 80 and 443. jetty:jetty owns everything under /opt/jetty/ and /opt/shibboleth-idp

Logs and configs below, though the configs are mostly a copy/paste from the instructions. I've also configured logback per the instructions, but that seems to be working, so I won't include the config unless necessary.

I should also note that I have not begun configuring Shibboleth. As I understand, I should still at least be able to run this command and get some results:

root@shib:/opt/shibboleth-idp/bin# ./status.sh 
(http://localhost/idp/status) Server returned HTTP response code: 503 for URL: http://localhost/idp/status

Before even navigating to any URL, I get this in /opt/jetty/jetty-base/log/jetty.log upon starting the jetty service:

    19:55:27.983 - WARN [org.eclipse.jetty.webapp.WebAppContext:531] - Failed startup of context o.e.j.w.WebAppContext@23a5fd2{/idp,[file:///opt/jetty/jetty-base/tmp/jetty-0.0.0.0-80-idp.war-_idp-any-1528794646328615543.dir/webinf/, jar:file:///opt/shibboleth-idp/war/idp.war!/],UNAVAILABLE}{/opt/shibboleth-idp/war/idp.war}
    org.eclipse.jetty.util.MultiException: Multiple exceptions
            at org.eclipse.jetty.annotations.AnnotationConfiguration.scanForAnnotations(AnnotationConfiguration.java:539)
            at org.eclipse.jetty.annotations.AnnotationConfiguration.configure(AnnotationConfiguration.java:448)
            at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:496)
            at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1363)
            at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
            at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
            at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:522)
            at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
            at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
            at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
            at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
            at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
            at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
            at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
            at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
            at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
            at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
            at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
            at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
            at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
            at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
            at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
            at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
            at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
            at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
            at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
            at org.eclipse.jetty.server.Server.start(Server.java:422)
            at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
            at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
            at org.eclipse.jetty.server.Server.doStart(Server.java:389)
            at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
            at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1540)
            at java.security.AccessController.doPrivileged(Native Method)
            at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1462)
            Suppressed: java.lang.NullPointerException: null
                    at org.eclipse.jetty.util.MultiReleaseJarFile.<init>(MultiReleaseJarFile.java:92)
                    at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:926)
                    at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:890)
                    at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164)
                    at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:551)
                    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                    at java.lang.Thread.run(Thread.java:748)
            Suppressed: java.lang.NullPointerException: null
                    at org.eclipse.jetty.util.MultiReleaseJarFile.<init>(MultiReleaseJarFile.java:92)
                    at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:926)
                    at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:890)
                    at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164)
                    at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:551)
                    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                    at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.NullPointerException: null
            at org.eclipse.jetty.util.MultiReleaseJarFile.<init>(MultiReleaseJarFile.java:92)
            at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:926)
            at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:890)
            at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:164)
            at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:551)
            at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
            at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
            at java.lang.Thread.run(Thread.java:748)
    19:55:27.989 - INFO [org.eclipse.jetty.server.AbstractConnector:278] - Started ServerConnector@16f7c8c1{HTTP/1.1,[http/1.1]}{0.0.0.0:80}
    19:55:28.273 - INFO [org.eclipse.jetty.util.ssl.SslContextFactory:290] - x509=X509@1e683a3e(1,h=[example.com],w=[example.com]) for SslContextFactory@2053d869(file:///opt/shibboleth-idp/credentials/idp-browser.p12,null)
    19:55:28.335 - INFO [org.eclipse.jetty.server.AbstractConnector:278] - Started ServerConnector@2f0a87b3{SSL,[ssl, http/1.1]}{0.0.0.0:443}
    19:55:28.335 - INFO [org.eclipse.jetty.server.Server:414] - Started @4819ms

/etc/default/jetty

JETTY_HOME=/opt/jetty/jetty-distribution
JETTY_BASE=/opt/jetty/jetty-base
JETTY_LOGS=/var/log/jetty

/opt/jetty/jetty-base/start.ini

# To disable the warning message, comment the following line
--module=home-base-warning

# ---------------------------------------
# Module: ext
--module=ext

# ---------------------------------------
# Module: resources
--module=resources

# ---------------------------------------
# Module: server
--module=server

# ---------------------------------------
# Module: http
--module=http

# ---------------------------------------
# Module: deploy
--module=deploy

# ---------------------------------------
# Module: jsp
--module=jsp

# ---------------------------------------
# Module: websocket
--module=websocket

# ---------------------------------------
# Module: jstl
--module=jstl

# ---------------------------------------
# Module: annotations
--module=annotations

# ---------------------------------------
# Module: logging
--module=logging

# ---------------------------------------
# Module: requestlog
--module=requestlog

# ---------------------------------------
# Module: servlets
--module=servlets

# ---------------------------------------
# Module: plus
--module=plus

# ---------------------------------------
# Mwdule: http-forwarded
--module=http-forwarded

--exec
--skip-file-validation=ssl
-XX:+UseG1GC
-Xmx1500m
-Djava.io.tmpdir=tmp
-Djava.library.path=libsetuid-linux
-Didp.home=/opt/shibboleth-idp
-Djava.security.egd=file:/dev/urandom

/opt/jetty/jetty-base/webapps/idp.xml

<Configure class="org.eclipse.jetty.webapp.WebAppContext">
  <Set name="war"><SystemProperty name="idp.home"/>/war/idp.war</Set>
  <Set name="contextPath">/idp</Set>
  <Set name="extractWAR">false</Set>
  <Set name="copyWebDir">false</Set>
  <Set name="copyWebInf">true</Set>
</Configure>

score 0 · Accepted Answer · answered Dec 07 '17 at 20:44

I managed to solve the problem today. After some more research, this Jetty bug seemed related to my issue. The patch didn't make the Jetty 9.3.22 release, so I built the latest 9.3.x branch from source and I'm using that now.

So, that is:

git clone https://github.com/eclipse/jetty.project.git
cd jetty.project
git checkout --track origin/jetty-9.3.x
mvn clean install

After building, copy and extract ./jetty-distribution/target/jetty-distribution-9.3.23-SNAPSHOT.tar.gz to your desired location.

Shibboleth IdP won't start with Jetty 9.3

1 Answers1