Hello I'm working with flink-MSK-Hudi architecture and I want to ingest data in my AWS Glue catalog. Currently If I try to ingest the data in a S3 bucket in hudi format in worked good, the problem is when I set the hive properties in the hudi sink When I create the source and sink there is no problem, the problem is when I insert the data in the sink.
The sink code I use is:
CREATE TABLE IF NOT EXISTS sink_table (
Col1 String,
Col2 String,
Col3 timestamp,
primary key (Col1) not enforced
)
WITH (
'connector' = 'hudi',
'write.tasks' = '4',
'path' = '<Path in S3 where hudi table must be saved>',
'table.type'= 'MERGE_ON_READ',
'hive_sync.enable' = 'true',
'hive_sync.auto_create_db'= 'true',
'hive_sync.db' = '<Database that already exists>',
'hive_sync.table'= 'TSTTable'
);
INSERT INTO sink_table
SELECT
*
from Source_table;
Can someone help me? I just want to see my hudi table in my Athena catalog.