Hudi DeltaStreamer error on `User class threw exception: java.lang.NullPointerException` at `...SchemaRegistryProvider.fetchSchemaFromRegistry`

Question

If you run into this error:

23/04/03 16:19:49 INFO Client:
     client token: N/A
     diagnostics: User class threw exception: java.lang.NullPointerException
    at org.apache.hudi.utilities.schema.SchemaRegistryProvider.fetchSchemaFromRegistry(SchemaRegistryProvider.java:82)
    at org.apache.hudi.utilities.schema.SchemaRegistryProvider.getSchema(SchemaRegistryProvider.java:100)
    at org.apache.hudi.utilities.schema.SchemaRegistryProvider.getSourceSchema(SchemaRegistryProvider.java:107)
    at org.apache.hudi.utilities.schema.SchemaProviderWithPostProcessor.getSourceSchema(SchemaProviderWithPostProcessor.java:42)
    at org.apache.hudi.utilities.deltastreamer.DeltaSync.registerAvroSchemas(DeltaSync.java:900)
    at org.apache.hudi.utilities.deltastreamer.DeltaSync.<init>(DeltaSync.java:238)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.<init>(HoodieDeltaStreamer.java:680)
...
23/04/03 16:19:49 ERROR Client: Application diagnostics message: User class threw exception: java.lang.NullPointerException
    at org.apache.hudi.utilities.schema.SchemaRegistryProvider.fetchSchemaFromRegistry(SchemaRegistryProvider.java:82)

I tried using the Apicurio Registry with the Core Registry API (Version 2), ending in /apis/registry/v2. My /etc/hudi/conf/hudi-defaults.conf file had hoodie.deltastreamer.schemaprovider.registry.url=https://apicurio.staging.zp-int.com/apis/registry/v2/ids/contentIds/815/ in it.

Answer 1

Use the Confluent Schema Registry API (Version 7) found in Apicurio Registry /apis/ccompat/v7. I changed my /etc/hudi/conf/hudi-defaults.conf to reference the new endpoint hoodie.deltastreamer.schemaprovider.registry.url=https://apicurio.staging.zp-int.com/apis/ccompat/v7/subjects/dbingestion_debezium.my_table-value/versions/latest and it worked.