How do I expose non-HTTP, TCP services in Kubernetes?

Question

I'm running a Kubernetes cluster in a public cloud (Azure/AWS/Google Cloud), and I have some non-HTTP services I'd like to expose for users.

For HTTP services, I'd typically use an Ingress resource to expose that service publicly through an addressable DNS entry.

For non-HTTP, TCP-based services (e.g, a database such as PostgreSQL) how should I expose these for public consumption?

I considered using NodePort services, but this requires the nodes themselves to be publicly accessible (relying on kube-proxy to route to the appropriate node). I'd prefer to avoid this if possible.

LoadBalancer services seem like another option, though I don't want to create a dedicated cloud load balancer for each TCP service I want to expose.

I'm aware that the NGINX Ingress controller supports exposing TCP and UDP services, but that seems to require a static definition of the services you'd like to expose. For my use case, these services are being dynamically created and destroyed, so it's not possible to define these service mappings upfront in a static ConfigMap.

Related discussion: [Kubernetes: Routing non HTTP Request via Ingress to Container](https://stackoverflow.com/q/49439899/320399) — blong, Aug 04 '22 at 17:07

score 1 · Answer 1 · answered Oct 12 '19 at 02:58

For non-HTTP, TCP-based services (e.g, a database such as PostgreSQL) how should I expose these for public consumption?

Well, that depends on how you expect the ultimate user to address those Services? As you pointed out, with an Ingress, it is possible to use virtual hosting to route all requests to the same Ingress controller, and then use the Host: header to dispatch within the cluster.

With a TCP service, such as PostgreSQL, there is no such header. So, you would necessarily have to have either an IP based mechanism, or assign each one a dedicated port on your Internet-facing IP

If your clients are IPv6 aware, assigning each Service a dedicated IP address is absolutely reasonable, given the absolutely massive IP space that IPv6 offers. But otherwise, you have two knobs to turn: the IP and the port.

From there, how you get those connections routed within your cluster to the right Service is going to depend on how you solved the first problem

Thanks for your answer. I understand that each service exposes its own unique port. My problem is that the current Kubernetes service types don't seem appropriate for my use-case. `NodePort`s require opening up the ports on nodes publicly which isn't ideal from a security point of view. LoadBalancer per-service is expensive. Assuming I can route correctly to the host, which Kubernetes service should I use to make those publicly connectable, particularly when these are being brought up and down dynamically? — cjheppell, Oct 21 '19 at 16:13

score 1 · Answer 2 · answered Oct 29 '19 at 22:31

Maybe this workflow can help:

(I make the assumption that the cloud provider is AWS)

AWS Console: Create a segregated VPC and create your Kubernetes ec2 instances (or autoscaling group) disabling the creation of public IP. This makes it impossible to access the instance from the Internet, you still can access through the private IP (ex, 172.30.1.10) via a Site 2 Site VPN or through a secondary ec2 instance in the same VPC with Public IP.
Kubernetes: Create a service with a Fixed NodePort (eg 35432 for Postgres).
AWS console: create a Classic o Layer 4 Loadblancer inside the same VPC of your nodes, in the Listeners Tab open the port 35432 (and other ports that you might need) pointing to one or all of your nodes via a "Target Group". There is no charge in the number of ports.

At this point, I don't know how to automate the update of the current living nodes in the Load Balancer's Target Group, this maybe could be an issue with Autoscaling features, if any... Maybe a Cron job with a bash script pulling info from AWS API and update the Target Group?

Thanks - this is helpful. Though obviously this isn't a generic solution for any Kubernetes environment and is tightly tied to the cloud provider of choice. Perhaps an [Operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) might be a good fit for a solution that hides the details of the cloud provider of choice? — cjheppell, Nov 21 '19 at 14:15

score 1 · Answer 3 · answered May 25 '22 at 09:39

A couple of years late to reply, but here is what I did for a similar case.

If you don't want to use LoadBalancer at all, then the other option is NodePort as you mention. To make it externally addressable, you can have the pod associate a static IP to its node when it comes up. For example in AWS EC2 you can have an elastic IP (or static external IP in GCP) and associate it when the postgresql pod comes up in its PodSpec using initContainers or a separate container:

  initContainers:
  - name: eip
    image: docker.io/amazon/aws-cli:2.7.1
    command:
    - /bin/bash
    - -xec
    - |
      INSTANCE_ID="$(curl http://169.254.169.254/latest/meta-data/instance-id)"
      aws ec2 associate-address --allocation-id "$EIP_ALLOCATION_ID" --instance-id "$INSTANCE_ID"
    env:
    - name: EIP_ALLOCATION_ID
      value: <your elastic IP allocation ID>
    - name: AWS_REGION
      value: <region>
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: aws-secret
          key: accessKey
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: aws-secret
          key: secretKey

Here AWS access key and secret key are assumed to be installed in aws-secret. Above example uses environment variables but to be more secure you can instead mount on a volume and read in the script body and unset after use.

To alleviate the security concerns related to ports, one option can be to add only a node or two to the security group exposing the port, and use nodeSelector for the pod to stick it to only those nodes. Alternatively you can use "aws ec2 modify-instance-attribute" in the container body above to add the security group to just the node running the pod. Below is a more complete example for AWS EC2 that handles a node having multiple network interfaces:

  containers:
  - name: eip-sg
    image: docker.io/amazon/aws-cli:2.7.1
    command:
    - /bin/bash
    - -xec
    - |
      INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
      PRIVATE_IP="$(curl http://169.254.169.254/latest/meta-data/local-ipv4)"
      for iface in $(curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/); do
        if curl "http://169.254.169.254/latest/meta-data/network/interfaces/macs/$iface/local-ipv4s" | grep -q "$PRIVATE_IP"; then
          INTERFACE_FOR_IP="$(curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/$iface/interface-id)"
        fi
      done
      if [ -z "$INTERFACE_FOR_IP" ]; then
        aws ec2 associate-address --allocation-id "$EIP_ALLOCATION_ID" --instance-id "$INSTANCE_ID"
      else
        aws ec2 associate-address --allocation-id "$EIP_ALLOCATION_ID" --network-interface-id "$INTERFACE_FOR_IP" --private-ip-address "$PRIVATE_IP"
      fi
      aws ec2 modify-instance-attribute --instance-id "$INSTANCE_ID" --groups $FULL_SECURITY_GROUPS
      tail -f -s 10 /dev/null
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/bash
          - -ec
          - |
            INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
            aws ec2 modify-instance-attribute --instance-id "$INSTANCE_ID" --groups $DEFAULT_SECURITY_GROUPS
    env:
    - name: EIP_ALLOCATION_ID
      value: <your elastic IP allocation ID>
    - name: DEFAULT_SECURITY_GROUPS
      value: "<sg1> <sg2>"
    - name: FULL_SECURITY_GROUPS
      value: "<sg1> <sg2> <sg3>"
    - name: AWS_REGION
      value: <region>
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: aws-secret
          key: accessKey
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: aws-secret
          key: secretKey

You may also need to set the deployment strategy to be Recreate instead of the default of RollingUpdate otherwise the lifecycle.preStop may get invoked after the container command in case the deployment is restarted using kubectl rollout restart deployment ....

How do I expose non-HTTP, TCP services in Kubernetes?

3 Answers3