1

I wanted to slightly modify Confluent's Git repo Dockerfile to have in my Confluent Connect page mongoDB and Snowflake connections. Everything runs ok but I don't see them in the portal.

Should docker-compose.yml be modified as well?

Original code:

FROM confluentinc/cp-server-connect-base:5.5.1

ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"

ARG CONNECTOR_NAME
RUN confluent-hub install --no-prompt confluentinc/${CONNECTOR_NAME}:5.5.0

Moded code:

FROM confluentinc/cp-server-connect-base:5.5.1

ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"

ARG CONNECTOR_NAME
RUN confluent-hub install --no-prompt confluentinc/${CONNECTOR_NAME}:5.5.0 \
   && confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:1.2.0 \
   && confluent-hub install --no-prompt snowflakeinc/snowflake-kafka-connector:1.4.3
marcin2x4
  • 1,321
  • 2
  • 18
  • 44

1 Answers1

2

I think you can try to do the following.

  1. Modify your Dockerfile:
FROM confluentinc/cp-server-connect-base:5.5.1

ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"

RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:1.2.0 \
   && confluent-hub install --no-prompt snowflakeinc/snowflake-kafka-connector:1.4.3

Since you need to install only mongodb and snowflake connectors.

  1. Use your custom image in docker-compose.yml:
...
  connect:
    # image: cnfldemos/cp-server-connect-datagen:0.3.2-5.5.0
    build: .
    hostname: connect
    container_name: connect
...
Iskuskov Alexander
  • 4,077
  • 3
  • 23
  • 38
  • First part done but I have no idea how to create docker-compose file. – marcin2x4 Sep 19 '20 at 18:43
  • 1
    Add lines from here https://github.com/meddlin/cpat/blob/9c510c785e1c2b8a944702c740eeb084965be4ac/docs/event-stream-layer/development-environment/docker-compose.yml#L57-L59 – Iskuskov Alexander Sep 19 '20 at 19:56
  • You mean only lines 57-59? – marcin2x4 Sep 19 '20 at 20:09
  • 1
    I found instruction for your case: https://github.com/meddlin/cpat/blob/9c510c785e1c2b8a944702c740eeb084965be4ac/docs/event-stream-layer/notes/confluent-getting-started.md#use-custom-kafka-image-in-kafka-docker-compose – Iskuskov Alexander Sep 19 '20 at 21:59
  • 1
    So you can build you own kafka connect image with tag (e.g. my-custom-image:1.0.0) and then use it in docker-compose. Or directly tell to docker-compose to build image (see my answer) – Iskuskov Alexander Sep 19 '20 at 22:01
  • We are almost there Alexander! I see both connectors in control center of Confluent's ;) – marcin2x4 Sep 19 '20 at 23:55
  • 1
    Intresting thing is that when I added `&& confluent-hub install --no-prompt snowflakeinc/snowflake-kafka-connector:1.4.3` line to your Dockerfile file everything worked. When I did same on previous files not all components got installed. Not that it does matter but worth mentioning. – marcin2x4 Sep 21 '20 at 18:18