1

I am messing aroung with Kafka Streams handled by the K8s. It goes more or less fine so far, yet weird behaviour is observed on the test environment:

[Consumer clientId=dbe-livestream-kafka-streams-77185a88-71a7-40cd-8774-aeecc04054e1-StreamThread-1-consumer, groupId=dbe-livestream-kafka-streams] We received an assignment [_livestream.dbe.tradingcore.sporteventmappings-table-0, _livestream.dbe.tradingcore.sporteventmappings-table-2, _livestream.dbe.tradingcore.sporteventmappings-table-4, _livestream.dbe.tradingcore.sporteventmappings-table-6, livestream.dbe.tennis.results-table-0, livestream.dbe.tennis.results-table-2, livestream.dbe.tennis.results-table-4, livestream.dbe.tennis.results-table-6, _livestream.dbe.betmanager.sporteventmappings-table-0, _livestream.dbe.betmanager.sporteventmappings-table-2, _livestream.dbe.betmanager.sporteventmappings-table-4, _livestream.dbe.betmanager.sporteventmappings-table-6] that doesn't match our current subscription Subscribe(_livestream.dbe.betmanager.sporteventmappings-table|_livestream.dbe.trading_states|_livestream.dbe.tradingcore.sporteventmappings-table|livestream.dbe.tennis.markets|livestream.dbe.tennis.markets-table); it is likely that the subscription has changed since we joined the group. Will try re-join the group with current subscription

As far as I understand, internal state somehow got broken, and Stream's source of truth conflicts with the broker/zookeeper's one. This behaviour never terminates: I just let it hang for few days beingh busy with another stuff, and still it's the, reported at the WARN level. More than that: no ERRORs were reported for this time.

I did not change nothing; did not deploy new instances; did not manipulate Kafka brokers in any way that might affect abovementioned Kafka Streams app. Any ideas what's wrong?

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Zazaeil
  • 3,900
  • 2
  • 14
  • 31

1 Answers1

4

The error message itself indicates, that something is wrong with your subscription. This may happen if you have two Kafka Streams instances using the same application.id, but both don't subscribe to the exact same topics.

In your case, the subscription does not contain livestream.dbe.tennis.results-table but corresponding partitions are assigned.

Note, that Kafka Streams requires that all instances with the same application.id are required to execute the exact same Topology and thus subscribe to the exact same topics.

krizajb
  • 1,715
  • 3
  • 30
  • 43
Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • I see the message, but it was just a `WARN`, saying that KS gonna try to rejoin. Should such an error be fixed by the KS generally or I experience an expected behavior? – Zazaeil Apr 04 '20 at 08:31
  • Well, KS tries to be resilient against the issue (so trying to rejoin is not unexpected). But I don't see that there is a bug that need to be fixed atm (given the provided data). It seems more like a user error, that different applications use the same `application.id`. There must be some app with the same `application.id` that subscribes to `livestream.dbe.tennis.results-table` -- hence, the correct "fix" is to let each app use its' own `application.id`. – Matthias J. Sax Apr 05 '20 at 21:11
  • I am having the same issue in a JUnit test with Embedded Kafka and Spring Cloud Stream Kafka Streams Binder. There is no "second application" running and I am cleaning up temporary folder for Zookeeper and Kafka before each run. Still if I have num.stream.threads set to anything other than 1 I get that error in 6 runs from 10, sometimes it starts and runs reasonably well (still throwing the same warnings but seemingly not affecting test results). – Sergey Shcherbakov May 17 '21 at 19:15
  • Maybe a bug in Spring? (Spring is doing a lot of weird stuff, and does not always use Kafka Streams "correctly"...) – Matthias J. Sax May 18 '21 at 15:16