1

There's a need for my team to use old version of StreamSet, version 2.5. But there is some important processors in version 3.8 we want to include in the old environment, namely the JDBC Meta Data Processor.

What have been done is below:

  1. Create a sample custom processor project with StreamSets version 2.5.1.1

    mvn archetype:generate -DarchetypeGroupId=com.streamsets -DarchetypeArtifactId=streamsets-datacollector-stage-lib-tutorial -DarchetypeVersion=2.5.1.1 -DinteractiveMode=true

  2. Modify the code based on JDBC Meta Data classes in StreamSets 3.8, with reference library in POM to StreamSets 3.8:

2.1 streamsets-datacollector-api

2.2 streamsets-datacollector-jdbc-protolib

  1. Package and upload the customized jar to external library of StreamSets UI.

  2. Upload the dependency jars to StreamSets UI:

4.1 streamsets-datacollector-api-3.8.jar

4.2 streamsets-datacollector-jdbc-protolib-3.8.jar

Everything is find except that once I uploaded

streamsets-datacollector-jdbc-protolib-3.8.jar

and restart StreamSets, there's error in the log:

java.lang.NoClassDefFoundError: com/streamsets/pipeline/api/base/configurablestage/DPushSource
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at com.streamsets.pipeline.BlackListURLClassLoader.findClass(BlackListURLClassLoader.java:77)
at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:341)
at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:323)
at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:325)
at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:192)
at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:66)
at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:48)
at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:66)
at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:44)
at com.streamsets.datacollector.main.Main$1.run(Main.java:101)
at com.streamsets.datacollector.main.Main$1.run(Main.java:98)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at com.streamsets.datacollector.main.Main.doMain(Main.java:98)
at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:308)

I noticed this class exists in API-3.8 jar and the class in API-2.5 exists but with a different package path,

API-3.8 com/streamsets/pipeline/api/base/configurablestage/DPushSource

API-2.5 com/streamsets/pipeline/configurablestage/DPushSource

Any hints I am missing to make it work? Thanks.

Felix
  • 269
  • 1
  • 11

1 Answers1

0

Needless to say, there are many fixes and enhancements since StreamSets Data Collector 2.5.0, and I would strongly recommend you upgrade.

Having said that, since DPushSource is specific to the origin, and you're focused on a processor, the easiest way forward is to simply remove the JDBC origin. Delete everything under jdbc-protolib/src/main/java/com/streamsets/pipeline/stage/origin/jdbc and try again.

metadaddy
  • 4,234
  • 1
  • 22
  • 46
  • Thanks for your advice, but I tried removing everything under origin/jdbc as you mentioned, there are many other dependencies missing from other part of protolib. – Felix Sep 24 '19 at 07:55
  • Yeah - there's a big gap between 2.5.0 and 3.8.0. I would upgrade if you possibly can. It's going to be very difficult to backport the JDBC Metadata Processor. – metadaddy Sep 24 '19 at 15:13