5

We have a docker image and a corresponding yaml file for the deployment using kubernetes. The application we have built is in scala with akka-http. We have used akka-cluster. We have a particular variable(seed-nodes in our case - akka cluster) in the configuration file which is used in our application code that uses the pod ip. But, we will not get the pod ip unless the deployment is done. How should we go about tackling the issue? Will environment variables help, if yes how?

More specifically, Once the docker images is deployed in the container in a pod, and when the container starts, the pod ip is already assigned. So, can we programmatically or otherwise configure the pod ips in our code or config file before the process starts in the container?

For reference, this is our configuration file : 

akka {
  actor {
    provider = "akka.cluster.ClusterActorRefProvider"
  }    
  remote {
    log-remote-lifecycle-events = off
    netty.tcp {
      hostname = "127.0.0.1"
      port = 0
    }
  }
  cluster {
    seed-nodes = [
      "akka.tcp://our-system@127.0.0.1:3000",
      "akka.tcp://our-system@127.0.0.1:3001",
    ],
    metrics {
      enabled = off
    }
  }    
}
service {
  validateTokenService {
     ml.pubkey.path = "<filePath>"
  }
  ml_repository {
    url = <url address>
  }
  replication.factor = 3
  http.service.interface = "0.0.0.0"
  http.service.port = 8080
}

In the above file, instead of having akka.remote.netty.tcp.hostname as "127.0.0.1", we need to have the pod-ip. So, that we can use it in the seed nodes as : 

seed-nodes = [
          "akka.tcp://our-system@hostname:3000",
          "akka.tcp://our-system@hostname:3001",
        ],

How can we do so? Thanks in advance.

Dreams
  • 5,854
  • 9
  • 48
  • 71
Nagireddy Hanisha
  • 1,290
  • 4
  • 17
  • 39
  • 1
    Check out this great blogpost http://paoloambrosio.net/2016/12/14/akka-location-transparency.html and its related GitHub demo project https://github.com/paoloambrosio/experiments/tree/akka-location-transparency . A solution is to run a discovery and export the seed hostnames during your docker image boot. – Stefano Bonetti Mar 01 '17 at 17:05

2 Answers2

3

I have tried something very similar. What you can do is use stateful sets Kubernetes-Stateful sets. Stateful sets have a naming convention and the pods will be named accordingly - you can read more about that in the link. This way, you can hard-code the values for the seed-nodes, since you know how the pods are going to be named. But, they have a drawback, stateful sets don't yet support rolling update(primarily because they are designed that way.) This article has a great explanation step by step : stateful set - akka clustering This article explains everything together - so posting a link instead of copying the steps here. Hope this helps

Dreams
  • 5,854
  • 9
  • 48
  • 71
  • Seems like the link is what I was looking for. Will try the approach in the link. – Nagireddy Hanisha Mar 14 '17 at 09:10
  • 1
    Considering this is 2017, and akka cluster is implemented in such a strange way .. that you basically have to hack your away around to make it work on kubernetes, and even if you do that, you don't get that cool rolling updates feature. Does this look wrong only to me ? – gerasalus Apr 01 '17 at 14:57
  • @gerasalus - Yes, it does seem weird, but I think it's because of the way stateful sets are designed. There seems to be some discussion going on around the same : https://github.com/kubernetes/kubernetes/issues/37566 – Dreams Apr 01 '17 at 15:35
  • 1
    @Tarun yes I agree. But I was referring more to the Akka cluster design. I also found a project on github which seems to use zookeeper for seed node logic, maybe that would be less painful that this stateful set approach with akka – gerasalus Apr 01 '17 at 19:31
  • @gerasalus - ohh, didn't know about it, can you share the link of the github project please? Would like to have a look at it. – Dreams Apr 03 '17 at 04:15
  • 1
    @Tarun https://github.com/sclasen/akka-zk-cluster-seed. Tried that today with couple of akka cluster nodes, seems to be working. Now just have to figure out what to do with removing nodes on scale down or other event.s – gerasalus Apr 03 '17 at 18:57
  • @gerasalus - Thanks a lot for sharing that :) Maybe you can add that as an answer as well! It would be really beneficial. – Dreams Apr 04 '17 at 04:07
  • @DenisMikhaylov - Thanks for the suggestion. Would surely explore it :) – Dreams Jun 08 '17 at 05:18
3

As of 2018 the ideal way to initialize a Akka Cluster within kubernetes is the akka-management.

It uses the Kubernetes API to fetch a list of all running pods and bootstrap the cluster. There's no need to configure any seed nodes.

It does also provice a readiness check that waits until the pod has joined the cluster. And a alive check.

Aki
  • 1,644
  • 12
  • 24