3

Can I terraform using the aws_msk_configuration the topics and their partition counts and other properties so I can disable automatic topic creation on the broker?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Archimedes Trajano
  • 35,625
  • 19
  • 175
  • 265

1 Answers1

1

You don't need an MSK specific provider for this.

There already is a Kafka Provider that should work with any kafka server, and uses internal Golang Kafka AdminClient commands (Sarama) to provision topics.

https://github.com/Mongey/terraform-provider-kafka

e.g https://github.com/Mongey/terraform-provider-kafka/issues/154#issuecomment-725454597

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • 3
    The Kafka provider requries a connection to kafka. So it's not part of the initial terraform. Second Kafka is not exposed to the Internet so this would require terraforming an EC2 instance with terraform. – Archimedes Trajano Sep 22 '22 at 12:46
  • But if Kafka is exposed to the internet, this approach works, right? – Vivere Nov 18 '22 at 12:32
  • 1
    You don't need an internet connection. You only need TCP access to the cluster. (VPC whitelist your local subnet). Or yes, an EC2 instance, Github actions, whatever. If you can't already connect to kafka with producer/consumer, then no it won't work, but then what's the point of having a Kafka cluster you can't connect to? It's "not part of initial terraform" because Terraform is built entirely on a plugin model, called providers – OneCricketeer Nov 18 '22 at 14:54
  • 1
    @OneCricketeer I think you misunderstood "not part of initial terraform". The point is not what is part of "core" terraform or not, but in this case what you can provision as part of a single terraform apply. Just like with provisioning Kubernetes resources with terraform hsa to be done _after_ EKS creation, the Terraform code for creating Kafka topics must run _after_ the Kafka cluster has been created in the first place. "initial terraform" would be the cluster-creating code in this scenario. – Marcello Romani Feb 28 '23 at 16:49
  • @MarcelloRomani And that is what MSK already does. OP was asking to create topics, not a Kafka cluster – OneCricketeer Feb 28 '23 at 17:25
  • I'm actually asking to have the topic creation as part of the initial terraform as per my comment in response to the answer. – Archimedes Trajano Apr 22 '23 at 04:03
  • I'm not sure I understand what you mean by "initial terraform". The MSK configuration cannot create any topic. But, Terraform reads all tf files in a directory. If you use the MSK resource to create a cluster, then gets its bootstrap servers as output, then you can pass that along to the kafka topic resource of the other provider. That could be done in the same file, or have `cluster.tf` and `topics.tf` and terraform makes a dependency graph if one depends on the other – OneCricketeer Apr 22 '23 at 12:40