0

Am I correct in assuming that the moment we rename an application, the semantics of the Kafka operator will completely change and might end up reading from the "initialOffset" by the application code ?

How is the semantics maintained for a definition of "application name" ?

Does every deploy of the application code result in a new application or it is simply using the @ApplicationAnnotation(name="") instance to define this meaning ?

brusli
  • 79
  • 9

2 Answers2

2

You can always launch the application by using -originalAppId the operator should continue from where the original app left-off. If you are using kafka 0.9 operator and you launch the application with same name, you can set the initialOffset to "application_or_latest" or "application_or_earliest", so the operator should continue from offsets that are processed in last run. The difference is if you specify -originalAppId the offsets are restored from checkpoint while the other one store offsets in kafka itself.

Siyuan Hua
  • 31
  • 2
1

You can launch an application from its previous state by using the -originalAppId parameter and provide the yarn application id from its previous run's checkpointed state, it should apply to all operator within the dag including the kafka input operator. You can also provide a new name for the application using attribute dt.attr.APPLICATION_NAME.

eg: launch pi-demo-3.4.0-incubating-SNAPSHOT.apa -originalAppId application_1459879799578_8727 -Ddt.attr.APPLICATION_NAME="pidemo v201"

ashwin111
  • 146
  • 1
  • 4