My Google Dataflow Job is keep getting failed while creating "Kafka-Bigquery" dataflow with AWS MSK public access bootstrap server. Is there any way to solve this issue?
I tried accessing public access bootstrap server by disabling unauthenticated access and enabling IAM based access, and also updated security groups to access from anywhere. also created a topic using EC2 instance terminal by accessing through private connection string. In GCP, I created a dataflow using Kafka-Bigquery template and passed public-access-bootstrap-connection-string, and topic created in EC2 also passed bigquery table id. Expectation: Job creates successfully and table is populated based on kafka message sent to msk server.
Can I connect AWS MSK (Kafka) with Google Dataflow (Kafka-Bigquery) Job (as consumer)?
