Unable to provide a Coder for org.apache.hadoop.hbase.client.Mutation using HBaseIO with FlinkRunner

Question

I am running into an issue that "Unable to provide a Coder for org.apache.hadoop.hbase.client.Mutation." using HbaseIO with FlinkRunner. The Exception is below:

Exception in thread "main" java.lang.IllegalStateException: Unable to return a default Coder for ParDo(HBaseProfile)/ParMultiDo(HBaseProfile).output [PCollection]. Correct one of the following root causes:
No Coder has been manually specified;  you may do so using .setCoder().

Inferring a Coder from the CoderRegistry failed: Unable to provide a Coder for org.apache.hadoop.hbase.client.Mutation.
  Building a Coder using a registered CoderProvider failed.
  See suppressed exceptions for detailed failures.
  Using the default output Coder from the producing PTransform failed: PTransform.getOutputCoder called.
    at org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
    at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:278)
    at org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:115)
    at org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:190)
    at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:536)
    at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
    at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:370)
    at ac.cn.iie.process.ProfileProcess.process(ProfileProcess.java:91)
    at ac.cn.iie.Bootstrap.main(Bootstrap.java:25)

I have using maven-shade-plugin to package my jar:

            <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer
                                    implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>ac.cn.iie.Bootstrap</mainClass>
                            </transformer>
                            <transformer
                                    implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                <resource>reference.conf</resource>
                            </transformer>
                        </transformers>
                        <relocations>
                            <relocation>
                                <pattern>org.codehaus.plexus.util</pattern>
                                <shadedPattern>org.shaded.plexus.util</shadedPattern>
                                <excludes>
                                    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                                    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                                </excludes>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>

Does anyone know the reason caused？

K Fred · Answer 1 · 2018-11-08T03:02:45.480

1

I have solved my issue by registering Beam CoderProvider manually. The code is below:

CoderRegistry coderRegistry = pipeline.getCoderRegistry();
coderRegistry.registerCoderProvider(
    new HBaseCoderProviderRegistrar().getCoderProviders().get(0));

The HBaseCoderProviderRegistrar() provide a CoderProvider list by getCoderProviders() method and the 0 index of list is HBaseMutationCoder. Because my code use the HBaseIO.write(), just need to register the Mutation Coder.

edited Nov 08 '18 at 03:02

answered Nov 08 '18 at 02:54

K Fred

81
4

Could you share more code please? I have the same issue. And nothing else helps. I dont understand what "to register a coder" means. – sg_rs Jul 19 '22 at 17:28
This one helped me so much. Thank you – sg_rs Jul 21 '22 at 14:13

score 0 · Answer 2 · answered Nov 08 '18 at 18:30

Note that the HBaseCoderProviderRegistrar already registers the HBaseMutationCoder for the Mutation type automatically already.

Using the maven-shade plugin without handling service files contained in META-INF/ inside your output jar is a common pitfall for users. You want to add the ServicesResourceTransformer to your transformers list, as so:

<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>

See Apache documentation and ServiceLoader javadoc for more details. There are many other questions like this on StackOverflow.

score 0 · Answer 3 · answered Aug 09 '21 at 02:57

0

Make the POJO class serializable. public class CustomerEntity implements Serializable {

private String id;

private String name;

answered Aug 09 '21 at 02:57

Prakash Dutta

81
9

score 0 · Answer 4 · answered Jul 21 '22 at 15:43

0

For me the answer of @K Fred helped. But I Have also discovered that just updating all apache.beam dependencies to the newest version solves the problem too.

answered Jul 21 '22 at 15:43

sg_rs

411
3
13

Unable to provide a Coder for org.apache.hadoop.hbase.client.Mutation using HBaseIO with FlinkRunner

4 Answers4