46

I have a Kubernetes cluster running on Google Compute Engine and I would like to assign static IP addresses to my external services (type: LoadBalancer). I am unsure about whether this is possible at the moment or not. I found the following sources on that topic:

  • Kubernetes Service Documentation lets you define an external IP address, but it fails with cannot unmarshal object into Go value of type []v1.LoadBalancerIngress
  • The publicIPs field seems to let me specify external IPs, but it doesn't seem to work either
  • This Github issue states that what I'm trying to do is not supported yet, but will be in Kubernetes v1.1
  • The clusterIP field also lets me specify an IP address, but fails with "provided IP is not in the valid range"

I feel like the usage of static IPs is quite important when setting up web services. Am I missing something here? I'd be very grateful if somebody could enlighten me here!

EDIT: For clarification: I am not using Container Engine, I set up a cluster myself using the official installation instructions for Compute Engine. All IP addresses associated with my k8s services are marked as "ephemeral", which means recreating a kubernetes service may lead to a different external IP address (which is why I need them to be static).

Community
  • 1
  • 1
Marco Lamina
  • 3,326
  • 4
  • 22
  • 22
  • What exactly did you do to solve this? Like you, I am afraid that if IP changes, I'll have to change the godaddy records to point to new IP address each time the deployment is recreated. – Srikanth Nov 01 '16 at 19:19
  • 1
    Use the `loadBalancerIP` field in your service declaration, as described in the current documentation: http://kubernetes.io/docs/user-guide/services/#type-loadbalancer – Marco Lamina Nov 04 '16 at 21:09

4 Answers4

72

TL;DR Google Container Engine running Kubernetes v1.1 supports loadBalancerIP just mark the auto-assigned IP as static first.

Kubernetes v1.1 supports externalIPs:

apiVersion: v1
kind: Service
spec:
  type: LoadBalancer
  loadBalancerIP: 10.10.10.10
  ...

So far there isn't a really good consistent documentation on how to use it on GCE. What is sure is that this IP must first be one of your pre-allocated static IPs.

The cross-region load balancing documentation is mostly for Compute Engine and not Kubernetes/Container Engine, but it's still useful especially the part "Configure the load balancing service".

If you just create a Kubernetes LoadBalancer on GCE, it will create a network Compute Engine > Network > Network load balancing > Forwarding Rule pointing to a target pool made of your machines on your cluster (normally only those running the Pods matching the service selector). It looks like deleting a namespace doesn't nicely clean-up the those created rules.

Update

It is actually now supported (even though under documented):

  1. Check that you're running Kubernetes 1.1 or later (under GKE edit your cluster and check "Node version")
  2. Allocate static IPs under Networking > External IP addresses, either:
    • Deploy once without loadBalancerIP, wait until you've an external IP allocated when you run kubectl get svc, and look up that IP in the list on that page and change those from Ephemeral to Static.
    • Click "Reserver a static address" regional in the region of your cluster, attached to None.
  3. Edit your LoadBalancer to have loadBalancerIP=10.10.10.10 as above (adapt to the IP that was given to you by Google).

Now if you delete your LoadBalancer or even your namespace, it'll preserve that IP address upon re-reploying on that cluster.

Update 2016-11-14

See also Kubernetes article describing how to set up a static IP for single or multiple domains on Kubernetes.

Wernight
  • 36,122
  • 25
  • 118
  • 131
  • Thank you -- `loadBalancerIP` works great. Perhaps they corrected a bug since this posting. – Mark Shust at M.academy Dec 03 '15 at 12:05
  • In 1.1 following the new procedure, loadBalancerIP works great. Is there a similar technique for adding HTTP/HTTPS load balancers? – TimK Feb 18 '16 at 23:31
  • @GNN HTTP/HTTPS has nothing to do with the IP. More accurately, you'd just have a LB service with a static IP, and that service would just have 80 and 443 ports (redirecting probably to a pod's 80 and 443 port). – Wernight Feb 19 '16 at 09:53
  • 1
    After spending hours reading google documentation trying to figure this out - i am so glad i found this post. good job. – IaMaCuP May 26 '16 at 19:28
  • This was helpful. The k8s documentation is confusing as it implies that the `loadBalancerIP` is the not the _ingress_ address of the load balancer, but it appears that it is. – jwadsack Aug 04 '16 at 22:52
  • @Wernight So, once the external ip is allocated to the loadbalancer, and that's marked as not-ephemeral, you can go ahead and point to that IP in godaddy? Even when that deployment is recreated, the ip holds? Like the OP's concerns, I am afraid that if the IP changes, the godaddy records have to be updated and will take time for dns caches to update resulting in my website being down in between... – Srikanth Nov 01 '16 at 19:15
  • @Srikanth Yes you're correct. At least on Kubernetes clusters supporting it, like GKE. – Wernight Nov 14 '16 at 14:17
  • The Kubernetes article link mentioned in your update is not available as of today. – hazardous Sep 25 '17 at 11:09
  • @hazardous Updated the link even though it's not the same page as the original is gone in history. – Wernight Sep 27 '17 at 08:18
15

Kubernetes v1.1 will make a few changes.

First, all load-balancers in GCE will get static IPs. This allows us to simulate "update" operations that GCE does not support.

Second, https://github.com/kubernetes/kubernetes/pull/13005 proposes a new field to explicitly set the IP of a load balancer.

Note though that your "ephemeral" IP is yours as long as your Service exists. This is roughly akin to what AWS does with ELB names (randomly assigned, yours until you release it).

publicIPs (or deprecatedPublicIPs in v1) will be replaced with externalIPs with very similar semantics. These are "unmanaged" IPs - kubernetes will not establish a load-balancer using them, but it will accept traffic for them.

clusterIP is an in-cluster address and generally is not available outside of the cluster or "project" or VPC (in GCE or AWS terms)

Tim Hockin
  • 3,567
  • 13
  • 18
  • 1
    Thanks for the thorough explanation :) So what is your suggestion if I want to delete + recreate a service and ensure that it keeps its external IP address? Is this possible with ephemeral IPs in v1.0.x? Also I'm quite excited for v1.1, is there a release date (or a rough estimate)? – Marco Lamina Sep 02 '15 at 07:30
  • 1
    Why do you need to delete a Service and recreate it with the same load-balancer IP? The short answer is that this isn't supported yet. It will hopefully be supported in v1.1 (mid-to-late October). There are some hacks you can do, if it is REALLY important to you, but all hacks come with costs :) – Tim Hockin Sep 02 '15 at 17:55
  • 2
    Well, I have the DNS entry for my public endpoint pointing to that IP address, so I want to make sure it won't change in case I have to change the k8s service object! I really thought this would be a common use case? :) – Marco Lamina Sep 03 '15 at 05:56
  • 1
    As mentioned, the fact that a change to a Service object resulted in an IP change is a bug and is rectified in v1.1. It is a common case - bugs suck :) – Tim Hockin Sep 03 '15 at 20:42
  • @TimHockin Plz correct me if I am wrong. When I create a service & mark it as not-ephemeral in Google Container Engine, the ip will persist across restarts & deletes. So, I can create such services for each of my clients (subdomains), such that each godaddy subdomain points to that particular service's external IP. Is this the standard way it's done or am I being ridiculous? Also, marking the ip as not-ephemeral will result in GCE billing me for static IP address? And, using the LoadBalancer for the service will result in Google charging me extra for it, right? – Srikanth Nov 01 '16 at 19:48
1

If you are running on Google Container Engine, and are using type: LoadBalancer, then Google Cloud Platform should have made a Network Load Balancer for you with a static IP address that will route to this service. You don't need to specify any IP addresses.

To find the network load balancer's IP, run:

gcloud compute forwarding-rules list --project "YOUR-PROJECT-ID"

You should also be able to run:

kubectl get services

Which will return both the cluster and external IP and port for your services.

Sandeep Dinesh
  • 2,035
  • 19
  • 19
  • I am not using Container Engine, I set up a cluster myself using the official installation instructions for Compute Engine. The forwarding rules get created, but the IP addresses associated with them are all marked as "ephemeral", which means recreating a kubernetes service may lead to a different external IP address. – Marco Lamina Aug 30 '15 at 10:08
  • In general you should never need to "recreate a service". There was a bug in v1.0 that caused some types of updates to release the IP address. This is fixed in v1.1. – Tim Hockin Sep 18 '15 at 21:56
  • @TimHockin Never recreate a service may be nice in some cases, but until there is a deployment solution that allows to apply only the changes it's a big issue. Even more if you think that each time you have to re-create your LoadBalancer service, you'll have to update your DNS and wait until it propagates. – Wernight Nov 20 '15 at 15:30
  • I'm not sure what "only apply the changes" means, but take a look at `kubectl apply` in v1.1 release. – Tim Hockin Nov 23 '15 at 03:51
  • @TimHockin For example if you `LoadBalancer` should point to a different set of labels or if you want to use the new Deployment method that recreated all services/pods, or similar cases. In all those cases you'd end up re-creating the `LoadBalancer` service and end up having a new "static" IP. `loadBalancerIP` should solve this in v1.1. – Wernight Dec 04 '15 at 15:40
  • @TimHockin & Others, I am also ready to reserve a static IP some way and want to use it in service specification spec: type: "LoadBalancer" loadBalancerIP: "107.178.xxx.xxx" Can this be part of future kubernetes release? – Raj Rajen Mar 08 '16 at 01:34
  • Raj, `loadBalancerIP` is already implemented, for platforms that support it... Is that what you're asking? – Tim Hockin Mar 15 '16 at 05:10
1

On my side I confirm that the loadBalancerIP field allows to specify a static IP address for a GKE LoadBalancer, using Kubernetes 1.18.2:

apiVersion: v1
kind: Service
spec:
  ...
  type: LoadBalancer
  loadBalancerIP: 10.136.0.49

In addition, command below returns the load balancer IP for a given project:

gcloud compute forwarding-rules list --project <PROJECT_IS>
NAME                              REGION       IP_ADDRESS   IP_PROTOCOL  TARGET
aaba14cac607a4413813bebfb7eaaf8a  us-central1  10.136.0.49  TCP          us-central1/backendServices/aaba14cac607a4413813bebfb7eaaf8a

And that command proves it works fine:

kubectl get service <svc-name> 
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
<svc-name>    LoadBalancer   10.136.27.201   10.136.0.49   4040:30754/TCP   57s
Fabrice Jammes
  • 2,275
  • 1
  • 26
  • 39