running Kafka Connect 4.1.1 on DC/OS using the confluent community package. How can we upload or add our jdbc driver to the remote cluster?
Update: It's a package installed DC/OS catalog, which is a mesos framework, running docker images.
running Kafka Connect 4.1.1 on DC/OS using the confluent community package. How can we upload or add our jdbc driver to the remote cluster?
Update: It's a package installed DC/OS catalog, which is a mesos framework, running docker images.
Update
Script borrowed from here (thanks to @rmoff)
It's an example of overriding the Docker CMD
with a bash script to download and extract the REST API source connector.
bash -c 'echo Installing unzip… && \
curl -so unzip.deb http://ftp.br.debian.org/debian/pool/main/u/unzip/unzip_6.0-16+deb8u3_amd64.deb && \
dpkg -i unzip.deb && \
echo Downloading connector… && \
curl -so kafka-connect-rest.zip https://storage.googleapis.com/rmoff-connectors/kafka-connect-rest.zip && \
mkdir -p /u01/connectors/ && \
unzip -j kafka-connect-rest.zip -d /u01/connectors/kafka-connect-rest && \
echo Launching Connect… && \
/etc/confluent/docker/run'
You'll need to build your own Docker images and publish them to a resolvable Docker Registry for your Mesos cluster, and then edit the Mesos Service to pull these images instead of the Confluent one.
For example, in your Dockerfiles, you would have
ADD http://somepath.com/someJDBC-driver.jar /usr/share/java/kafka-connect-jdbc
Or curl
rather than ADD
, as shown in the Confluent docs (because it needs to extract that .tar.gz
file).
FROM confluentinc/cp-kafka-connect
ENV MYSQL_DRIVER_VERSION 5.1.39
RUN curl -k -SL "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-${MYSQL_DRIVER_VERSION}.tar.gz" \
| tar -xzf - -C /usr/share/java/kafka-connect-jdbc/ --strip-components=1 mysql-connector-java-5.1.39/mysql-connector-java-${MYSQL_DRIVER_VERSION}-bin.jar
You can also use confluent-hub install
to add other connectors that aren't JDBC JAR files