7

Does kafka connect start new connector and its tasks within kafka connect process? or a new JVM process will be forked.

If it starts plugin within kafka connect process, then I need set kafka connect JVM heap size via KAFKA_CONNECT_JVM_HEAP_OPT (using confluent docker image). Then the problem is, if I start many tasks or many connectors, they will share the JVM heap, so it is hard to decide the heap size of kafka connect.

If for each connector, kafka connect starts them in a new JVM process, how can I set the heap size for them?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Xiang Zhang
  • 2,831
  • 20
  • 40

2 Answers2

8

Kafka Connect has basic support for multi-tenancy. Specifically, you are able to bundle several connector instances within the same Connect worker.

Each Connect worker always maps to a single JVM instance. A request to start a new connector does not result into spawning a new JVM instance. But Connect workers with the same group.id form a cluster of Connect workers. Then, connector tasks are distributed among the workers in the Connect cluster.

A Connect worker's heap size can be easily set using:

export KAFKA_HEAP_OPTS="-Xms256M -Xmx2G" (this example uses the default values)

or, when a docker image is used, by setting:

-e CONNECT_KAFKA_HEAP_OPTS="-Xms256M -Xmx2G" (again this example uses the default values)

Connect workers can be scaled horizontally. Adding more workers in a Connect cluster adds memory and computing resources to your deployment. If you need to apply a more specific and tight memory budget to your Connect deployment, you might chose to group specific connectors to each Connect cluster, or even in some cases deploy one connector instance per Connect cluster.

Konstantine Karantasis
  • 1,855
  • 14
  • 15
3

All tasks share the memory space within one worker's host OS, whether that's a container doesn't really matter (other than the fact without JVM flags on the process inside container, it's limited even further)

You "add memory" to your Connect cluster by adding more workers. You prevent OOM errors by increasing topic partitions, Connector tasks, reducing poll/batch amounts, and reducing the overall amount of data each worker needs to read.

The environment variable for Connect's heap settings is KAFKA_HEAP_OPTS, and you can add more JVM flags from KAFKA_OPTS

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks, but I mean JVM heap size, do all works running in one kafka connect container share the same JVM heap? not only host OS – Xiang Zhang May 31 '18 at 13:44
  • When I say memory here, I do mean heap – OneCricketeer May 31 '18 at 13:46
  • They all run under a single Java process... I'm not sure why you think they'd have individual heaps. And if so, if one connector died, that might mean the container overall would die? Because the Docker exec process is the Java command to run Connect – OneCricketeer May 31 '18 at 13:48
  • I'm asking because I want to be sure they are in the same process. There are other containers they folk process to keep container itself robust. Knowing this will help me to decide the JVM heap size. – Xiang Zhang May 31 '18 at 13:56
  • 1
    That's somewhat against the philosophy of "microservices", but okay. I mean, Connect launches a web server to accept messages, then threads for a Consumers and Producers, but it's not an entirely new Java process being executed – OneCricketeer May 31 '18 at 13:57