5

I'm specifying dataflow runner in my beamSql program below :

DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
    options.setStagingLocation("gs://gcpbucket/staging");
    options.setTempLocation("gs://gcpbucket/tmp");
    options.setProject("beta-19xxxx");
    options.setRunner(DataflowRunner.class);
    Pipeline p = Pipeline.create(options);

But I'm getting below exception : 

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
at my.proj.StarterPipeline.main(StarterPipeline.java:34)Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@782830e, only URLClassLoaders are supported.
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:43)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:262)

can anyone help me out understand what exactly is the issue ?

Andrew Nguonly
  • 2,258
  • 1
  • 17
  • 23
Nagesh Singh Chauhan
  • 703
  • 10
  • 24
  • 1
    Are you using Java 9? Java 9 is currently not supported by Apache Beam. Take a look at this Stack Overflow post: https://stackoverflow.com/questions/48292491/java-dataflow-unable-to-use-classloader-to-detect-classpath-elements. This is the same issue. Recommend downgrading to Java 8 for now. – Andrew Nguonly Mar 12 '18 at 14:35
  • Well yes !! Java 9 was the problem, I downgraded it to Java 8 and it worked. Thanks Andrew – Nagesh Singh Chauhan Mar 13 '18 at 02:54
  • 1
    I got the similar issue but I'm using java 8. Can someone tell me what may be the reason of this issue. – Pardeep Sharma Jan 08 '19 at 11:20

3 Answers3

6

Downgrading java 9 to java 8 did the trick.

Nagesh Singh Chauhan
  • 703
  • 10
  • 24
1

This problem can happen when using Beam SDK with JVM > 8.

Support for JVM 11 (or 9) is only experimental, starting from Beam SDK 2.12 and at least up to Beam SDK 2.16. This answer provides a bit more context: https://stackoverflow.com/a/57710742/1046584

Regarding this specific stack trace, it seems related to this issue: https://issues.apache.org/jira/browse/BEAM-3718

Luís Bianchin
  • 2,327
  • 1
  • 28
  • 36
1

For me the fix was different. In my pom.xml, below dependency was set to 2.20 while apache beam was set to 2.19

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
  <version>2.20</version>
</dependency>

After changing it to same as beam version, build was working fine.

Net Net, just make sure all the beam related sdks are set to same version. Use variable for the same, like ${beamVersion} so you don't end up facing strange errors.

JD-V
  • 3,336
  • 1
  • 17
  • 20