How can I insert data from prestoDB to Kafka topic?

Question

I can get data from kafka topic from presto like below properties:

/etc/kafka/mytable.json

{
    "tableName": "mytable",
    "schemaName": "mykafka",
    "topicName": "mykafka.mytable",
    "message": {
        "dataFormat": "json",
        "fields": [
            {
                "name": "id",
                "mapping": "id",
                "type": "INTEGER"
            },
            {
                "name": "name",
                "mapping": "name",
                "type": "VARCHAR"
            }
        ]
    }

it works when getting data. select * from kafka.mykafka."mytable"

But when i want to insert data it gives this error:

insert into kafka.mykafka."mytable"(id,name) values(111,'presto db test');


Query 20211007_053816_00016_x6wzd, FAILED, 1 node
Splits: 2 total, 1 done (50.00%)
0:00 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20211007_053816_00016_x6wzd failed: unknown data format 'dummy'

How can I handle this?

thanks in advance

Last I checked, Presto is meant for querying, not inserting data. Plus, it doesn't implement a Kafka Producer at all — OneCricketeer, Oct 07 '21 at 15:55

score 0 · Answer 1 · answered Mar 05 '22 at 15:06

While Presto is mainly meant for querying the data from different data sources using appropriate connectors, it also seems possible to insert data using Presto. While this is straightforward for data sources such as MySQL, its little tricky in case of Kafka as there is encoding involved. Refer https://trino.io/docs/current/connector/kafka.html#kafka-inserts for more details.

How can I insert data from prestoDB to Kafka topic?

1 Answers1