0

I have a simple kafka streams app that utilizes interactive queries and uses RESTfull for instances communication.

The app works fine outside of docker with multiple streams instance. And the streams#allMetadata() returns a correct list of all running instances. However, once I run the app inside docker, the results of streams#allMetadata() is an empty array [].

Note: when running in docker, everything works except the discoverability of other instances. [I can interact with different streams instances and can get the data they are storing]

I've already looked at other posts here with the same issue, but mine seems a bit different other posts

  • I have the same APPLICATION_ID in all my instances.
  • The communication between Kafka and any individual instance is working. And Streams are working correctly [meaning BOOTSTRAP_SERVERS_CONFIG is configured correctly]
  • Data are stored correctly on all instances and I can interact with each instance directly through its IP[meaning APPLICATION_SERVER_CONFIG is configured correctly on each instance]
  • I am running the same version of Kafka and Streams v1.1.0.
  • I tried confluent and landoop Kafka docker images, same results.

What am I missing here?

J S
  • 175
  • 1
  • 4
  • 14
  • Did you double check the actual assignment? Does the application process data when running in docker? I am a little puzzled atm about what the issue could be. – Matthias J. Sax Jul 17 '18 at 16:44
  • We're using a dockerized streams app, so it's possible for sure. A couple of things to check: is the port configured in your `APPLICATION_SERVER_CONFIG` exposed externally by docker on the same port? Are there any cases where multiple streams apps run in different containers on the same hosts? – Kyle Fransham Jul 17 '18 at 21:20
  • 1
    @MatthiasJ.Sax : Yes I did check partition assignment, my partitions were assigned correctly and I can see the streaming internal topic getting created in Kafka with the correct data [aggregation]. That's why I was very confused. By the way, I found the root cause issue and posted the solution below. – J S Jul 17 '18 at 22:48
  • 1
    @KyleFransham: Yes I made sure the ports are assigned correctly, I was able to interact with each individual instance correctly. So it was not a port configuration issue. By the way. I found the root cause and posted its solution below. – J S Jul 17 '18 at 22:52

1 Answers1

2

I found the issue and resolved it.

The issue was caused by a missing library [ld-linux-x86-64.so.2] from my streaming app docker base image, that library is used by one of RocksDB's dependencies [librocksdbjni8077190960714261011.so] (go figure!).

The solution was to just use another base image that has the above library, so I changed my streaming app's base image from openjdk:8-jdk-alpine to anapsix/alpine-java:latest and everything works correctly now.

Note: when I had the above issue, I've noticed another symptom. My streaming app would lose connection to kafka broker sporadically. I would see every now and then a message on the log about "unable to reach the broker node...". This issue disappeared after the fix above. I have no idea how they correlate.

J S
  • 175
  • 1
  • 4
  • 14
  • Great you figured it out. I am still wondering what the connection of the wrong base image to the issue is? Did all threads die due to the missing library and thus, after partitions assignment happened metadata was empty? If yes, did you find this out via logs? – Matthias J. Sax Jul 18 '18 at 00:41
  • 1
    @MatthiasJ.Sax: That is correct, the assignment of tasks occurs initially, but then the tasks would shut down and I end up with empty metadata. Sometimes the logs would only show `Invalid transition attempted from state READY to state ABORTING_TRANSACTION` for each task. But again that was sporadic. – J S Jul 18 '18 at 01:06