Apache Spark isn't playing nice with Jersey dependency injection

Question

I'm trying to use the com.github.sps.metrics.metrics-opentsdb library to log metrics from my spark job to my OpenTSDB server. I'm running into an issue where I get a strange NPE down in the jersey code that deals with EncodingFilters.

Here is the exception I'm getting:

ERROR OpenTsdb: send to opentsdb endpoint failed
javax.ws.rs.ProcessingException: java.lang.NullPointerException
    at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:582)
    at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:564)
    at org.apache.cxf.jaxrs.client.WebClient.doResponse(WebClient.java:1148)
    at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1098)
    at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:894)
    at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)
    at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:428)
    at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1635)
    at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1630)
    at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.post(WebClient.java:1570)
    at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.post(InvocationBuilderImpl.java:145)
    at com.github.sps.metrics.opentsdb.OpenTsdb.sendHelper(OpenTsdb.java:176)
    at com.github.sps.metrics.opentsdb.OpenTsdb.send(OpenTsdb.java:162)
    at com.github.sps.metrics.opentsdb.OpenTsdb.send(OpenTsdb.java:138)
    at com.adobe.gto.gateway.common.metrics.Metrics$RunnableMetricRecorder.run(Metrics.java:220)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at org.glassfish.jersey.client.filter.EncodingFilter.getSupportedEncodings(EncodingFilter.java:110)
    at org.glassfish.jersey.client.filter.EncodingFilter.filter(EncodingFilter.java:84)
    at org.apache.cxf.jaxrs.client.spec.ClientRequestFilterInterceptor.handleMessage(ClientRequestFilterInterceptor.java:60)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
    at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:652)
    at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1097)`

I've looked at the code in EncodingFilter.class and see that the line I'm failing on is this:

List<ContentEncoder> encoders = serviceLocator.getAllServices(ContentEncoder.class);

In the class I see that serviceLocator isn't instantiated, instead I see this line at the top of the class:

@Inject
private ServiceLocator serviceLocator;

I initially thought it might be a bad dependency so I checked my mvn dependency:tree and made sure all the versions for cxf and jersey projects matched, so I don't think that's the issue. Now I'm thinking it has something to do specifically with Apache Spark, the reason I think that is because I have a separate application that uses the same metrics-opentsdb library and I never see the above exception, everything works correctly in my other application.

So, does anyone know what's going on with jersey dependency injection and spark? Has anyone dealt with this same problem and fixed it? Is the problem something else altogether?

Are you packaging your application into a "fat jar" (using Maven/Gradle's shadowing plugin)? If so - the issue is probably service files being overridden. — Tzach Zohar, Jan 13 '17 at 08:59
@TzachZohar, I am using maven's Shade plugin, but I experience the issue when running the app in eclipse too. I'm under the impression that while running it in eclipse I wouldn't have the problem of files being overridden. Is that not the case? — Hardy, Jan 13 '17 at 16:42
Umm.. you got me there - I would assume it would work in Eclipse too... — Tzach Zohar, Jan 13 '17 at 17:01

score 0 · Accepted Answer · answered Jan 13 '17 at 23:36

A co-worker pointed out that CXF and Jersey are competing products and suggested I remove CXF since the metrics-opentsdb library says it uses Jersey. So I re-wrote a portion of my app that was using CXF and added an <exclusions> section to one of my dependencies in my pom.xml and now everything seems to work fine.

Apache Spark isn't playing nice with Jersey dependency injection

1 Answers1