0

I need to publish the Big query table rows to Kafka in Avro format.

PCollection<TableRow> rows =
        pipeline
            .apply(
                "Read from BigQuery query",
                BigQueryIO.readTableRows().from(String.format("%s:%s.%s", project, dataset, table))
    
//How to convert rows to avro format?

rows.apply(KafkaIO.<Long, ???>write()
                .withBootstrapServers("kafka:29092")
                .withTopic("test")
                .withValueSerializer(KafkaAvorSerializer.class)
        );

How to convert TableRow to Avro format?

vkt
  • 1,401
  • 2
  • 20
  • 46

1 Answers1

1

Use MapElements

rows.apply(MapElements.via(new SimpleFunction<Tabelrows, GenericRecord>() {
  @Override
  public GenericRecord apply(Tabelrows input) {
    log.info("Parsing {} to Avro", input);
    return null; // TODO: Replace with Avro object
  }
});

If Tabelrows is a collection-type that you want to convert to many records, you can use FlatMapElements instead.

As for writing to Kafka, I wrote a simple example

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I am looking on how can we convert the tablerows to avro generic record? – vkt Dec 03 '22 at 16:10
  • Well, you mistyped Tabelrow, or haven't shown that class definition so I cannot answer that. Read the Javadoc for TableRows object?... Call getters. Create new a GenericRecord and call setters. Return it. That's not unique to Bigquery – OneCricketeer Dec 03 '22 at 17:32
  • TableRow is Big query table row format https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableRow.html. ... I am not sure how can I convert that table row to the generic record of Avro? – vkt Dec 04 '22 at 14:51
  • Your question shows `Tablerows` (with an s). I don't know what that is. But the linked class implements Map, so like I said, call `input.get("column")`...then read the Javadoc for GenericRecord, and you'll see it has set methods. I trust you're capable of figuring it out – OneCricketeer Dec 04 '22 at 15:07