1

I need to launch Kafka Rest Proxy on kubernetes. When I used docker-compose to start it on local machine with a local Kafka broker and zookeeper it went well and I got the last log output stating that Kafka Rest is running and waiting for requests. When I launch it on kubernetes I get only the following output:

===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ... 
===> Check if Kafka is healthy ...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-log4j12-    1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory[]
log4j:WARN No appenders could be found for logger (io.confluent.admin.utils.cli.KafkaReadyCommand).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

And then the pod exits with error code 1 and in in CrashLoopBackOff state.

For the environmental variables I've set up KAFKA_REST_BOOTSTRAP_SERVERS with format of PLAINTEXT://{host1},PLAINTEXT://{host2} and KAFKA_REST_LOG4J_ROOT_LOGLEVEL = DEBUG but it didn't make any additional output to appear.

Did anyone have the same problem?

fpiechowski
  • 517
  • 1
  • 7
  • 21
  • The error is saying that the log4j file isn't loaded properly. That's why you can't see logs or why changing the level didn't work – OneCricketeer Aug 15 '21 at 12:20
  • Hi @Mesayah, any progress? – Mikolaj S. Aug 16 '21 at 08:43
  • @Mikolaj Unfortunately, not. I'm running kafka-rest from the official Docker image so I expect that all log4j configs are present there. I've notices that when launching the image locally the SimpleLoggerFactory binding is used and when it is launched on remote kubernetes cluster the Log4jLoggerFactory binding isused. From what I've read in case of multiple bindings found on classpath the random one is used. – fpiechowski Aug 16 '21 at 09:17
  • Could you share `.yaml` configuration files that you are deploying to Kubernetes, so it will be easier to replicate the issue? Did you follow some tutorial, if yes could share it also? – Mikolaj S. Aug 16 '21 at 09:55
  • We're using fiaas for kubernetes deployments. You can see its config here: https://pastebin.com/imDg1sR2 I've also run the docker image built by CI and pushed to out docker registry and it was working locally and used SimpleLoggerFactory – fpiechowski Aug 16 '21 at 10:47
  • 1
    Besides that I was following official Kafka REST Proxy docs, and used the image I found in Confluent Platform docker prebuilt package. – fpiechowski Aug 16 '21 at 10:53
  • Your yaml there has no environment variables to actually configure the server, so that'd explain why it's crashing. Otherwise, I suggest trying older versions of the image since they've been making several changes to their Docker build process lately, and it's hard to follow it. You can try making an issue here, but I'm not sure if they monitor it https://github.com/confluentinc/kafka-rest-images/issues – OneCricketeer Aug 20 '21 at 12:53
  • @OneCricketeer I've built the project from sources and provided only the bootstrap servers property just like in the compose. Now it works. I sincerely think that prebuilt docker images are crap. – fpiechowski Aug 21 '21 at 07:58
  • I don't think you need to build from source (at least, not the master branch) since you can just get Confluent Platform as a whole or use APT/YUM to install the proxy, but it's still unclear how you're providing environment variables to the image to begin with – OneCricketeer Aug 21 '21 at 15:34
  • I have similar problem: https://stackoverflow.com/questions/69633706/confluent-kafka-rest-proxy-docker-confluentinc-cp-kafka-rest6-2-1-couldnt-st – Alex Pristenskiy Oct 25 '21 at 14:06

2 Answers2

0

There indeed seems to be an issue with the Log Config in latest versions of the Proxy. I had the same issue and reverted to 5.1.0, which gave me good log output and pointed out that it was missing the environment property KAFKA_REST_HOST_NAME.

Florian Albrecht
  • 2,266
  • 1
  • 20
  • 25
0

I had exactly the same problem with Kafka REST Proxy in Docker container and my reason was misspell in IP address and port, something like '192.168.100:14:9092' and after replacement to '192.168.100.14:9092' all started to work and I even saw logs, but failure logs was exactly like yours (no meaningful logs and no info why it was failing).

Alex Pristenskiy
  • 33
  • 2
  • 10