We are running a node service on docker (image node:lts-bullseye) We are currently using kafkajs, which is really easy to use, the problem is that sometimes after the Kafka rebalances and moves topics from one broker to another we can produce messages to the topic because of some OS level cache of the DNS, so every time this happens we need to reset the pods (that's what we found in our research), the solution seems to be to move to node-rdkafka, the problem there is that I manage to work with it locally but on the docker it does not work, I am subscribing to the "ready" event but it never happens, we also don't get any other events such as errors or logs.
nodejs version: 18-lts image: node:lts-bullseye (both for builder & runtime) kafka: AWS MSK kafka security protocol: sasl_ssl
both solutions to fix the problem with kafkajs or with node-rdkafka will be welcome.
edit:
seems to be a problem with the sasl configuration where it just hanging, my configuration for the producer looks like this:
{
"metadata.broker.list": "*********,*********",
"client.id": "*********",
"message.max.bytes": 15728640,
"retries": 5,
"security.protocol": "sasl_ssl",
"sasl.mechanism": "scram-sha-512",
"sasl.username": "*********",
"sasl.password": "*********"
}
edit:
found the current problem, apparently"scram-sha-512" needs to be in upper case for this library, only found out when passing callback to the connect function its weird because when giving a mechanism the broker does not support we do get an error event. now locally the server crushes with "segmentation failure" and on the remote it just crushes
edit:
here is the dockerfile:
FROM node:lts-bullseye AS builder
# apt-get for certificates
RUN apt-get update && apt-get install ca-certificates -q -y
# create the app folders
RUN mkdir -p /app
WORKDIR /app
# copy the package.json and package-lock.json files
COPY ./package.json ./package-lock.json /app/
# install the modules without copying the rest of the project, to benefit from layer caching
RUN npm install -q --unsafe-perm
# set prod env variables
ENV NODE_ENV=production
ENV PROD=true
# copy the rest of the project
COPY . /app
# set npm index
ARG VERSION
RUN npm version $VERSION
ARG SENTRY_AUTH_TOKEN
ENV SENTRY_AUTH_TOKEN=$SENTRY_AUTH_TOKEN
ARG COMMIT_HASH
ENV COMMIT_HASH=$COMMIT_HASH
ARG RUN_NUMBER
ENV RUN_NUMBER=$RUN_NUMBER
# build the project
RUN npm run build:prod
# set sentry release
RUN sh ./run-sentry.sh
FROM node:lts-bullseye AS runtime
# create folders for runtime
RUN mkdir -p /app
WORKDIR /app
# create user and give access
RUN adduser --disabled-login user
RUN chown -R user:user /app
RUN mkdir -p /var/log
RUN chown -R user:user /var/log
USER user
# copy ready files from builder
COPY --from=builder --chown=user:user /app /app
# set prod env variables
ENV NODE_ENV=production
ENV NODE_PATH=dist/
EXPOSE 8080
ENTRYPOINT ["sh", "/app/entrypoint.sh"]
