How to expose two apps/services over unique ports with k3d?

Question

*Cross-posted from k3d GitHub Discussion: https://github.com/rancher/k3d/discussions/690

I am attempting to expose two services over two ports. As an alternative, I'd also love to know how to expose them over the same port and use different routes. I've attempted a few articles and a lot of configurations. Let me know where I'm going wrong with the networking of k3d + k3s / kubernetes + traefik (+ klipper?)...

I posted an example: https://github.com/ericis/k3d-networking

The goal:

• Reach "app-1" on host over port 8080
• Reach "app-2" on host over port 8091

Steps

*See: files in repo

1. Configure k3d cluster and expose app ports to load balancer

   ports:
     # map localhost to loadbalancer
     - port: 8080:80
       nodeFilters:
         - loadbalancer
     # map localhost to loadbalancer
     - port: 8091:80
       nodeFilters:
         - loadbalancer
   

2. Deploy apps with "deployment.yaml" in Kubernetes and expose container ports

   ports:
     - containerPort: 80
   

3. Expose services within kubernetes. Here, I've tried two methods.

   • Using CLI

     $ kubectl create service clusterip app-1 --tcp=8080:80
     $ kubectl create service clusterip app-2 --tcp=8091:80
     

   • Using "service.yaml"

     spec:
       ports:
       - protocol: TCP
         # expose internally
         port: 8080
         # map to app
         targetPort: 80
       selector:
         run: app-1
     

4. Expose the services outside of kubernetes using "ingress.yaml"

   backend:
     service:
       name: app-1
       port:
         # expose from kubernetes
         number: 8080
   

Answer 1

You either have to use an ingress, or have to open ports on each individual node (k3d runs on docker, so you have to expose the docker ports)

Without opening a port during the creation of the k3d cluster, a nodeport service will not expose your app

k3d cluster create mycluster -p 8080:30080@agent[0]

For example, this would open an "outside" port (on your localhost) 8080 and map it to 30080 on the node - then you can use a NodePort service to actually connect the traffic from that port to your app:

apiVersion: v1
kind: Service
metadata:
 name: some-service
spec:
 ports:
 - protocol: TCP
   port: 80
   targetPort: some-port
   nodePort: 30080
 selector:
   app: pgadmin
 type: NodePort

You can also open ports on the server node like: k3d cluster create mycluster -p 8080:30080@server[0]

Your apps can get scheduled to run on whatever node, and if you force a pod to appear on a specific node (lets say you open a certain port on agent[0] and set up your .yaml files to work with that certain port), for some reason the local-path rancher storage-class just breaks and will not create a persistent volume for your claim. You kinda have to get lucky & have your pod get scheduled where you need it to. (if you find a way to schedule pods on specific nodes without the storage provisioner breaking, let me know)

You also can map a whole range of ports, like: k3d cluster create mycluster --servers 1 --agents 1 -p "30000-30100:30000-30100@server[0]" but be careful with the amount of ports you open, if you open too much, k3d will crash.

Using a load balancer - it's similar, you just have to open one port & map to to the load balancer.

k3d cluster create my-cluster --port 8080:80@loadbalancer

You then have to use an ingress, (or the traffic won't reach)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello
            port:
              number: 80

I also think that ingress will only route http & https traffic, https should be done on the port 443, supposedly you can map both port 80 and port 443, but I haven't been able to get that to work (I think that certificates need to be set up as well).