Custom HoodieRecordPayload for use in flink sql

Question

I am trying to use Apache Hudi with Flink sql by following Hudi's flink guide
The basics are working, but now I need to provide custom implementation of HoodieRecordPayload as suggested on this FAQ.
But when I am passing this config as shown in following listing, it doesn't work. Basically my custom class (MyHudiPoc.Poc) doesn't get picked and instead normal behaviour continues.


CREATE TABLE t1(
  uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
  name VARCHAR(10),
  age INT,
  ts TIMESTAMP(3),
  `partition` VARCHAR(20)
)
PARTITIONED BY (`partition`)
WITH (
  'connector' = 'hudi',
  'path' = '/tmp/hudi',
  'hoodie.compaction.payload.class' = 'MyHudiPoc.Poc', -- My custom class
  'hoodie.datasource.write.payload.class' = 'MyHudiPoc.Poc',  -- My custom class
  'write.payload.class' = 'MyHudiPoc.Poc',  -- My custom class
  'table.type' = 'MERGE_ON_READ'
);

INSERT INTO t1 VALUES
  ('id1','Danny',23,TIMESTAMP '1970-01-01 00:00:01','par1'),
  ('id2','Stephen',33,TIMESTAMP '1970-01-01 00:00:02','par1'),
  ('id3','Julian',53,TIMESTAMP '1970-01-01 00:00:03','par2'),
  ('id4','Fabian',31,TIMESTAMP '1970-01-01 00:00:04','par2'),
  ('id5','Sophia',18,TIMESTAMP '1970-01-01 00:00:05','par3'),
  ('id6','Emma',20,TIMESTAMP '1970-01-01 00:00:06','par3'),
  ('id7','Bob',44,TIMESTAMP '1970-01-01 00:00:07','par4'),
  ('id8','Han',56,TIMESTAMP '1970-01-01 00:00:08','par4');


insert into t1 values
  ('id1','Danny1',27,TIMESTAMP '1970-01-01 00:00:01','par1');

I even tried passing it through /etc/hudi/conf/hudi-default.conf

---
"hoodie.compaction.payload.class": MyHudiPoc.Poc
"hoodie.datasource.write.payload.class": MyHudiPoc.Poc
"write.payload.class": MyHudiPoc.Poc

I am also passing my custom jar while starting flink sql client.

/bin/sql-client.sh embedded \
    -j ../jars/hudi-flink1.15-bundle-0.12.1.jar \
    -j ./plugins/flink-s3-fs-hadoop-1.15.1.jar \
    -j ./plugins/parquet-hive-bundle-1.8.1.jar \
    -j ./plugins/flink-sql-connector-kafka-1.15.1.jar \
    -j my-hudi-poc-1.0-SNAPSHOT.jar \
    shell

I am able to pass my custom class in spark example but not in flink.
Tried with both COW and MOR type of tables.

Any idea what I am doing wrong?

See listing in the question.

Custom HoodieRecordPayload for use in flink sql

0 Answers0