4

Problem

I have two pods A and B running in a cluster on minikube, both have external IPs www.service-a.com and www.service-b.com. Both external IPs are accessible from outside.

I need A to be able to call B with it's external IP rather than its cluster DNS, that is A needs to use www.service-b.com rather than b.svc.cluster.local (which does work but I can't use it).

I set A to use hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet. If I spin up a NodeJS docker container manually, it indeed can connect and find it. However, A is still unable to connect to service-b.com. Am I using hostNetwork wrong? How can I configure my pod to connect to b in that way?

A's Deployment YAML

...
spec:
  replicas: 1
  selector:
    matchLabels:
      app: a-app
  template:
    metadata:
      labels:
        app: a-app
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
...

B's service YAML

...
spec:
  externalTrafficPolicy: Cluster
  type: LoadBalancer
  ports:
  - port: ...
    targetPort: ...
    protocol: TCP
    name: http
...

Background:

I'm using Minio (a local S3-like solution) and I need to presign the URLs to get and put objects. Minio's pods are running in the same cluster as my authentication pod which would generate the presigned urls. The presigned urls would be used from outside the cluster. Hence I can't sign the url with the cluster dns names like minio.svc.cluster.local because this URL would not be accessible from outside the cluster and replacing the host with my-minio.com and keeping the signature does not work because I guess minio signs the entire host and path. Hence I need to have my authentication pod connect to Minio's public facing my-minio.com instead which does not seem to work.

Math is Hard
  • 896
  • 1
  • 12
  • 24

2 Answers2

6

Regarding hostNetwork, it looks like you misunderstood it. Setting it to true means that Pod will have access to the host where it's running. In case of minikube it's VM and not your host where actual containers are running.

Also, i'm not sure how you expose your services to external world, but i suggest you to try Ingress for that.

Grigoriy Mikhalkin
  • 5,035
  • 1
  • 18
  • 36
  • I've been just using a LoadBalancer with externalTrafficPolicy turned on. With an ingress would the pod be able to connect to the other pod via the ingress? I'm pretty novice at Kube. – Math is Hard Nov 11 '20 at 20:52
  • Okay going to try that now. Will update here once I'm done – Math is Hard Nov 11 '20 at 20:58
  • I deployed an ingress and a nginx controller `NAMESPACE NAME HOSTS ADDRESS PORTS AGE my-namespace my-minio localhost localhost 80 16m` on my local machine, hitting localhost now directs to pod B but A is unable to hit `localhost` and get to pod B. – Math is Hard Nov 12 '20 at 02:14
  • A method to replace the Host header with `nginx.ingress.kubernetes.io/upstream-vhost` seem pretty promising but, just putting that annotation there in my ingress doesn't seem to work out of the box. Perhaps it's because I installed nginx through helm? – Math is Hard Nov 12 '20 at 03:53
  • @MathisHard Can you place another question, describing your issues with ingress/ingress controller? Seems to me like this is issue for another SO question. – acid_fuji Nov 12 '20 at 08:29
  • @MathisHard For minikube you could just run `kubectl get pods -n kube-system` to install ingress controller. Here is [tutorial on how to set and use ingress in minikube](https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/) – Grigoriy Mikhalkin Nov 12 '20 at 11:38
  • @MathisHard Also, can you update your question and add manifest for Ingress? – Grigoriy Mikhalkin Nov 12 '20 at 11:39
  • I have posted another question that involves the ingress solution to this particular problem: https://stackoverflow.com/questions/64815229/nginx-controller-kubernetes-need-to-change-host-header-within-ingress – Math is Hard Nov 13 '20 at 03:52
  • 1
    Actually got it to work turns out I shouldn't declare rewrite-target and ssl-redirect. Replacing upstream-vhost works as expected. Now my problem is solved :) – Math is Hard Nov 13 '20 at 08:02
2

As Grigoriy suggested, I used an ingress with nginx.ingress.kubernetes.io/upstream-vhost annotation to forward all requests into the cluster with Host: service-b to resolve my issue. Previously I had nginx.ingress.kubernetes.io/rewrite-target: /$1 which stripped the path from the request that caused a serious of issues, so I removed that. The details of how I got it working are here:

NGINX controller Kubernetes: need to change Host header within ingress

Math is Hard
  • 896
  • 1
  • 12
  • 24