1

if I already have Kafka running on premises, is Kafka Connect just a configuration on top of my existing Kafka, or does Kafka Connect require it's own Server/Environment separate from that of my existing Kafka?

Artanis
  • 561
  • 1
  • 7
  • 26

2 Answers2

2

Kafka Connect is part of Apache Kafka, but it runs as a separate process, called a Kafka Connect Worker. Except in a sandbox environment, you would usually deploy it on a separate machine/node from your Kafka brokers.

This diagram shows conceptually how it runs, separate from your brokers:

Diagram of Kafka Connect

You can run Kafka Connect on a single node, or as part of a cluster (for throughput and redundancy).

You can read more here about installation and configuration and architecture of Kafka Connect.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
0

Kafka Connect is its own configuration on top of your bootstrap-server's configuration.

For Kafka Connect you can choose between a standalone server or distributed connect servers and you'll have to update the corresponding properties file to point to your currently running Kafka server(s).

Look under {kafka-root}/config and you'll see enter image description here

You'll basically update connect-standalone or connect-distributed properties based on your need.

pacman
  • 1,061
  • 1
  • 17
  • 36
  • I'm confused because you say it's just configuration on top of the bootstrap server but then you also mention a Connect server and documentation talks about Connect clusters – Artanis Mar 29 '19 at 13:50
  • I edited my answer to make it more clear. You can run kafka-connect in standalone mode or distributed (cluster) mode based on your need on how much data you are processing using connect. – pacman Mar 29 '19 at 14:39
  • So am I right in saying: You need to provision a separate server(s) for Connect and they are configured to work with existing Kafka server. But bottom line is Connect requires it's own server(s) and it's not just a config file added to the existing kafka server(s). – Artanis Mar 29 '19 at 14:54
  • Yes, that sounds correct. You kick off Connect as a new process and probably a good idea to kick it off on its own server. However, you can get away with running this process on the same server where your bootstrap Kafka server is running depending on the resources the Connect process will use. Hope this helps. Cheers. – pacman Mar 29 '19 at 15:05