Using confluent-kafka python client in alpine container

Question

I'm trying to run a simple python app that communicates with kafka. I'm looking to use an alpine container for it. Here's my current dockerfile (it's not optimal... just trying to get things working for now).

FROM python:3.6-alpine
MAINTAINER Ashic Mahtab (ashic@live.com)

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update && apk --no-cache add librdkafka


COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt

COPY api /usr/src/app/api
COPY static /usr/src/app/static

CMD ["python", "api/index.py"]

The requirements file has confluent-kafka in it. The build fails with

OK: 8784 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  so:libcrypto.so.41 (missing):
    required by:
                 librdkafka-0.9.4-r1[so:libcrypto.so.41]
                 librdkafka-0.9.4-r1[so:libcrypto.so.41]
                 librdkafka-0.9.4-r1[so:libcrypto.so.41]
  so:libssl.so.43 (missing):
    required by:
                 librdkafka-0.9.4-r1[so:libssl.so.43]
                 librdkafka-0.9.4-r1[so:libssl.so.43]
                 librdkafka-0.9.4-r1[so:libssl.so.43]

My questions are a) is there a way to get this working without building inside the container? It'd be good enough if I could simply copy the library over to alpine. Or even if I could copy librdkafka over. b) If not, how can I get libssl and libcryto.so working?

For whoever using confluent-kafka v1.4.0. Don't use it.. Use v1.4.2. There is an issue that it doesn't contain requirements.txt. See [this](https://github.com/confluentinc/confluent-kafka-python/issues/830) post. — Louis Go, Jul 29 '21 at 09:25

score 4 · Answer 1 · answered Apr 28 '17 at 08:31

4

here a way to make it work:

FROM python:3.6-alpine
MAINTAINER Ashic Mahtab (ashic@live.com)

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \
    && apk upgrade --update-cache --available \
    && apk --no-cache add librdkafka

COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt

COPY api /usr/src/app/api
COPY static /usr/src/app/static

CMD ["python", "api/index.py"]

Here an explanation:

python:3.6-alpine image are based on alpine linux 3.4: Dockerfile

So you first need to correctly move to edge alpine branch: Edge
This is done by the line:

RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \
    && apk upgrade --update-cache --available \

Then you can install librdkafka and the dependencies.

answered Apr 28 '17 at 08:31

Gregoire

49
3

With ```RUN sed -i -e 's/v3\.4/edge/g' /etc/apk/repositories \ && apk upgrade --update-cache --available \ && apk --no-cache add alpine-sdk gcc librdkafka``` I get ```fatal error: librdkafka/rdkafka.h: No such file or directory``` – ashic Apr 28 '17 at 09:30
Usually, header files are in package called `-dev`, so try to install `librdkafka-dev` – Gregoire Apr 28 '17 at 11:56
For further, missing files, you can use this to find the corresponding package: https://pkgs.alpinelinux.org/contents – Gregoire Apr 28 '17 at 11:58
This will switch a stable alpine image to the "current development tree" (edge). This is fine for some experimentation but is risky in production and I would recommend against this. I highly suggest yuo add a disclaimer to this answer. – exhuma May 03 '21 at 12:34

score 4 · Answer 2 · answered Apr 30 '17 at 04:56

4

I maintain the ucalgary/python-librdkafka image that extends the official python:3.6-alpine image and installs librdkafka from its source releases. You're welcome to use the image, or take a look at the Dockerfile to see how it's built.

answered Apr 30 '17 at 04:56

King Chung Huang

5,026
28
24

You mention Python 3.6 in this answer, but looking at the Dockerfile I see Python 3.7 referenced. This is misleading – exhuma May 03 '21 at 12:36

score 0 · Answer 3 · answered Apr 28 '17 at 08:59

Since you're on edge anyway, you could also just use python3 from there:

% cat Dockerfile 
FROM alpine:edge
MAINTAINER Gerd Flaig <stackoverflow@gerd.flaig.name>

RUN apk update && apk add --no-cache python3 librdkafka && \
    python3 -m ensurepip && \
    rm -r /usr/lib/python*/ensurepip && \
    pip3 install --upgrade pip setuptools && \
    rm -r /root/.cache
% docker build .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM alpine:edge
edge: Pulling from library/alpine
71c5a0cc58e4: Pull complete
Digest: sha256:99588bc8883c955c157d18fc3eaa4a3c1400c223e6c7cabca5f600a3e9f8d5cd
Status: Downloaded newer image for alpine:edge
 ---> 8914de95a28d
Step 2/3 : MAINTAINER Gerd Flaig <stackoverflow@gerd.flaig.name>
 ---> Running in 658b056c4e16
 ---> a3b2485fabb0
Removing intermediate container 658b056c4e16
Step 3/3 : RUN apk update && apk add --no-cache python3 librdkafka &&     python3 -m ensurepip &&     rm -r /usr/lib/python*/ensurepip &&     pip3 install --upgrade pip setuptools &&     rm -r /root/.cache
 ---> Running in d7c3f3b30d89
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.5.0-4449-g8922925e5c [http://dl-cdn.alpinelinux.org/alpine/edge/main]
v3.5.0-4450-gb7b4122d6f [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 8348 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/16) Installing libressl2.5-libcrypto (2.5.3-r1)
(2/16) Installing libgcc (6.3.0-r3)
(3/16) Installing libressl2.5-libssl (2.5.3-r1)
(4/16) Installing libstdc++ (6.3.0-r3)
(5/16) Installing librdkafka (0.9.4-r1)
(6/16) Installing libbz2 (1.0.6-r5)
(7/16) Installing expat (2.2.0-r0)
(8/16) Installing libffi (3.2.1-r3)
(9/16) Installing gdbm (1.12-r0)
(10/16) Installing xz-libs (5.2.3-r0)
(11/16) Installing ncurses-terminfo-base (6.0-r7)
(12/16) Installing ncurses-terminfo (6.0-r7)
(13/16) Installing ncurses-libs (6.0-r7)
(14/16) Installing readline (6.3.008-r5)
(15/16) Installing sqlite-libs (3.18.0-r0)
(16/16) Installing python3 (3.6.1-r1)
Executing busybox-1.26.2-r0.trigger
OK: 75 MiB in 27 packages
Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages
Requirement already satisfied: pip in /usr/lib/python3.6/site-packages
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages
Collecting setuptools
  Downloading setuptools-35.0.2-py2.py3-none-any.whl (390kB)
Collecting packaging>=16.8 (from setuptools)
  Downloading packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools)
  Downloading appdirs-1.4.3-py2.py3-none-any.whl
Collecting six>=1.6.0 (from setuptools)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools)
  Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
Installing collected packages: pyparsing, six, packaging, appdirs, setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed appdirs-1.4.3 packaging-16.8 pyparsing-2.2.0 setuptools-35.0.2 six-1.10.0
 ---> cfb1033ceec0
Removing intermediate container d7c3f3b30d89
Successfully built cfb1033ceec0

librdkafka installs from edge, but pip installing confluent-kafka fails with the missing header file error mentioned above. — ashic, Apr 28 '17 at 09:33

score 0 · Answer 4 · answered Aug 08 '23 at 14:40

0

You can do this

RUN apk add librdkafka-dev --repository=https://dl-cdn.alpinelinux.org/alpine/edge/community

answered Aug 08 '23 at 14:40

Александр

1

Using confluent-kafka python client in alpine container

4 Answers4