Setting Up DB2 in Airflow

Question

I am trying to connect DB2 as a data connection for airflow which is residing in a Docker Container (realizing that this is not nativly supported). I am developing on a Mac

I have added the connectionas seen in the below screenshot where the URL is host:port/databse.

I am then going to the Data Profiling > Ad Hoc Query to try and test the connection and I get the below.

In order to make sure the drivers were available, I mounted the folder where the jdbc driver is located to /usr/local/airflow/drivers in the docker-compose file.

I also made sure to include the below packages in my requirements.txt as these were required when I query from a jupyter notebook.

sasl
thrift_sasl
jaydebeapi
jpype1
ibm_db
ibm_db_sa

I can't figure out what I'm missing.

I've been through:

Airflow documentation
Unable to setup a DB2 / DashDB JDBC Connection in Apache Airflow
Apache Airflow db2 Connection
Apache Airflow db2 Connection
https://github.com/puckel/docker-airflow
Lots of GitHub issues
and many more resources but can't find anything that solves this

Here is my current Dockerfile. As indicated in the comments, JVM isn't installed in the Dockerfile so that may be the issue.

# VERSION 1.10.1
# AUTHOR: Matthieu "Puckel_" Roisil
# DESCRIPTION: Basic Airflow container
# BUILD: docker build --rm -t puckel/docker-airflow .
# SOURCE: https://github.com/puckel/docker-airflow

FROM python:3.6-slim
LABEL maintainer="Puckel_"

# Never prompts the user for choices on installation/configuration of packages
ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux


# Airflow
ARG AIRFLOW_VERSION=1.10.1
ARG AIRFLOW_HOME=/usr/local/airflow
ARG AIRFLOW_DEPS=""
ARG PYTHON_DEPS=""
ENV AIRFLOW_GPL_UNIDECODE yes

# Define en_US.
ENV LANGUAGE en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LC_CTYPE en_US.UTF-8
ENV LC_MESSAGES en_US.UTF-8

# Java

RUN apt-get update && apt-get install -y openjdk-7-jre-headless wget \
    && apt-get clean 
ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64

RUN set -ex \
    && buildDeps=' \
        freetds-dev \
        libkrb5-dev \
        libsasl2-dev \
        libssl-dev \
        libffi-dev \
        libpq-dev \
        git \
    ' \
    && apt-get update -yqq \
    && apt-get upgrade -yqq \
    && apt-get install -yqq --no-install-recommends \
        $buildDeps \
        freetds-bin \
        build-essential \
        default-libmysqlclient-dev \
        apt-utils \
        curl \
        rsync \
        netcat \
        locales \
    && sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen \
    && locale-gen \
    && update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 \
    && useradd -ms /bin/bash -d ${AIRFLOW_HOME} airflow \
    && pip install -U pip setuptools wheel \
    && pip install pytz \
    && pip install pyOpenSSL \
    && pip install ndg-httpsclient \
    && pip install pyasn1 \
    && pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,ssh${AIRFLOW_DEPS:+,}${AIRFLOW_DEPS}]==${AIRFLOW_VERSION} \
    && pip install 'redis>=2.10.5,<3' \
    && if [ -n "${PYTHON_DEPS}" ]; then pip install ${PYTHON_DEPS}; fi \
    && apt-get purge --auto-remove -yqq $buildDeps \


  && apt-get autoremove -yqq --purge \
    && apt-get clean \
    && rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/man \
        /usr/share/doc \
        /usr/share/doc-base

COPY script/entrypoint.sh /entrypoint.sh
COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
COPY requirements.txt ${AIRFLOW_HOME}/requirements.txt

RUN pip install --upgrade pip && pip install -r requirements.txt
RUN chown -R airflow: ${AIRFLOW_HOME}

EXPOSE 8080 5555 8793

USER airflow
WORKDIR ${AIRFLOW_HOME}
ENTRYPOINT ["/entrypoint.sh"]
CMD ["webserver"] # set default arg for entrypoint

score 1 · Answer 1 · answered Nov 28 '18 at 04:01

1

The jpype1 module requires a JVM, accessible on your $PATH -- try installing one and try again.

answered Nov 28 '18 at 04:01

joebeeson

4,159
1
22
29

Do you have a resouce I could follow. Admittedly, this is not my strong suit. I think its found at usr/libexec/java_home but I may just be implementing things incorrectly. – Alex Nov 28 '18 at 17:26
Updated the original question to indicate my local development is being done on a macbook – Alex Nov 28 '18 at 17:33
I tried adding ENV JAVA_HOME /usr/libexec/java_home to my Dockerfile, but still received the same error – Alex Nov 28 '18 at 17:38
Is Java available in the Docker container? – joebeeson Nov 28 '18 at 18:00
Thats why I'm trying to figure out. I pulled the container from puckel/docker-airflow which, I guess I just assumed, should have Java given Airflow has a JDBC connection option. This may not be the case though – Alex Nov 28 '18 at 19:25
I do not see the Dockerfile defining a JVM. – joebeeson Nov 28 '18 at 19:31
Nor do I. I'm trying adding this now `RUN apt-get update \ && apt-get install -t jessie-backports --no-install-recommends -y openjdk-8-jre-headless \ && rm -rf /var/lib/apt/lists/* ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64` – Alex Nov 28 '18 at 19:33
Added my Dockerfile for reference – Alex Nov 28 '18 at 21:23

score 0 · Answer 2 · answered Nov 29 '18 at 20:41

Finally figured it out. Below is what I ended up using to get Java installed. then I just mounted the folder with my driver in it.

# Java
RUN mkdir -p /usr/share/man/man1 && \
    (echo "deb http://http.debian.net/debian stretch main" > /etc/apt/sources.list.d/backports.list) && \
    apt-get update -y \
    && apt-get install --no-install-recommends -y build-essential libkrb5-dev libsasl2-dev libffi-dev default-libmysqlclient-dev vim-tiny gosu krb5-user openjdk-8-jre openjdk-8-jdk-headless openjdk-8-jdk openjdk-8-jre-headless \
    && apt-get clean

RUN apt-get install unzip -y && \
    apt-get autoremove -y

Setting Up DB2 in Airflow

2 Answers2