3

I have a few libraries for doing IO, with large amounts of data, that use akka-http. Now. I have a Flink application, where I need to use those libraries.

How do I use a common actor-system across my flink app? I mean, AFAIU, flink apps will be distributed and the same pipeline could end up creating multiple actor systems on multiple JVMs. Also, the same JVM could be shared between multiple pipelines too, and in that case, having multiple actor systems would mean having a lot of threads.

Am I right in assuming that this is a problem? If so, how can I avoid that problem? More specifically, is there a way to share an actor system between the apps on the same JVM? I know Flink has a way to get the same execution context using the Async IO.

Bhashit Parikh
  • 3,121
  • 23
  • 24
  • You are right; creating actor-system instances in Operators will end up to having multiple actor-system instances in same JVM, and it's costly. I suggest you to use singleton pattern for instantiating your custom actor-system, per JVM. for example: `object AkkaProvider { lazy val instance = ActorSyste() }` ... But you should care about actor-path that you are using, because as you have mentioned, you can have multiple instnacces from an Operator and using a simple static name in that Operator to create actors (with same name) will fail. – Reza Same'ei Jul 15 '18 at 08:45
  • 1
    Plus one note; because Flink is using Akka internally, you should be aware from classloading problems because of unmatched akka-libs from your application and Flinks dependencies. I know that Flink 1.4 has changed it's classloading policy and mechanism for jobs that should solve the problem! but I've not experienced it yet. – Reza Same'ei Jul 15 '18 at 08:45

0 Answers0