0

AS per documentation on confluent https://docs.confluent.io/platform/current/connect/references/restapi.html we can use connect rest api to start/stop/manage kafka connect - just want to understand what is the preferable way when we are deploying our resource on cloud (in my case aws) where we have restricted UI access.

  1. We have all the application and kafka in private subnet.
  2. We can not use any public ip at all to expose the GUI or hit the REST Kafka connect API directly via bowser or POSTMAN.
  3. We can't do port forwarding as well due to access restriction.

Want to understand how people runs production grade Kafka with direct access restriction - some open source tool which we can use directly? anything creating and made opensource around REST Connect API? Kindly guide on this.

Thank you

Narendra Makwana
  • 1

1 Answers1

0

can't do port forwarding

Even with an ELB/ALB? That will be a necessary step to access the API at all

restricted UI access

Kafka Connect does not have a UI, so you don't need to worry about that. MSK Connect dashboard does, but that is only accessible when you login to the AWS Account.

can not use any public ip at all to expose the GUI or hit the REST Kafka connect API directly via bowser or POSTMAN

It definitely shouldn't be public, but you could use a SOCKS proxy to get around that limitation, as long as you have SSH access. I would suggest not using desktop/browser to interact with the API, though. You should keep your configurations as code; for example, Terraform an MSK Connector.

You could also use EKS and deploy Strimzi operator, for example, and manage Connect deployments as Kubernetes resources. Then, following GitOps-style, connectors are controlled with Git actions within a private subnet, rather than interact directly with any Connect HTTP endpoints.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245