0

On a GKE cluster, I have client and server pods, with a client service and a server service.

My server service is:

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    app: server-deployment
  ports:
    - port: 5000
      targetPort: 5000

When I access the client pod shell and run

nslookup server-cluster-ip-service

I get

Server:         server-IP
Address:        server-IP-Address

Name:   server-cluster-ip-service.default.svc.cluster.local
Address: IPAddress

** server can't find server-cluster-ip-service.svc.cluster.local: NXDOMAIN

** server can't find server-cluster-ip-service.cluster.local: NXDOMAIN


** server can't find server-cluster-ip-service.svc.cluster.local: NXDOMAIN
** server can't find server-cluster-ip-service.cluster.local: NXDOMAIN

** server can't find server-cluster-ip-service.us-central1-c.c.my-cluster.internal: NXDOMAIN

** server can't find server-cluster-ip-service.google.internal: NXDOMAIN

** server can't find server-cluster-ip-service.us-central1-c.c.my-cluster: NXDOMAIN

** server can't find server-cluster-ip-service.google.internal: NXDOMAIN

** server can't find server-cluster-ip-service.c.my-cluster.internal: NXDOMAIN
** server can't find server-cluster-ip-service.c.my-cluster.internal: NXDOMAIN

The service is running on port 5000, because when I set up a pod with busybox, I can curl from that pod like so:

curl server-cluster-ip-service:5000

And it returns json from my server.

After experimenting with what address to put in the fetch request in my client code, the only way I can get a 200 response is with this:

const getAllUsers = async () => {
    console.log("GETTING ALL USERS");
    const response = await fetch("server-cluster-ip-service.default.svc.cluster.local", {
  mode: 'cors',
  headers: {
    'Access-Control-Allow-Origin':'*'
  }
});
    const resp = await response
    console.log("RESPONSE", resp)
    const json = await response.json();

    setUsers(json);
  };

enter image description here

which returns not json and not apparently on port 5000,

whereas all attempts to query the service at port 5000 fail.

I have this running locally and it works fine. I have ruled out arm processor issues with my mac by building and pushing docker images in GKE from the cloud console. I am fairly confident this is a GKE issue, because the dns works locally, but why would it not work with GKE? I don't have any network policies I've set myself - could there be a node security group blocking access? I read about "shielding" as a node security policy configured at setup, but I don't know how to check if that's been configured?

Complete code below:

My server code is:

const express = require("express");
const bodyParser = require("body-parser");

var cors = require("cors");

const PORT = 5000;

const app = express();
app.use(cors());

app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());
app.use(express.static("public"));

app.listen(PORT, function () {
  console.log("listening on 5000");
});

app.get("/", (req, res) => {
  console.log("PROCESSING GET USERS REQUEST");
  const list = ["item1", "item2", "item3"];
  res.json(list);
});

My client code is:

import { useState, useEffect } from "react";

import "./editUser.css";

    function EditUser() {
      const [users, setUsers] = useState([]);
    
      const getAllUsers = async () => {
        console.log("GETTING ALL USERS");
        const response = await fetch("http://server-cluster-ip-service:5000");
        const json = await response.json();
    
        setUsers(json);
      };
    
      useEffect(() => {
        getAllUsers();
      }, []);
    
      return (
        <div className="App">
          <h1 data-position="header">Merr k8s testbed</h1>
    
          <section data-position="quotes">
            <h2>Console</h2>
            <ul>
              {users &&
                users.map((user) => (
                  <li>
                    <h3>{user}</h3>
                  </li>
                ))}
            </ul>
          </section>
        </div>
      );
    }
    
    export default EditUser;

My client-deployment.yaml is:

kind: Deployment
metadata:
  name: client-deployment
  labels:
    app: client-deployment
    component: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: client-deployment
  template:
    metadata: 
      labels:
        app: client-deployment
        component: web
    spec:
      containers:
        - name: client
          image: myDocker/k8s-client:latest
          ports:
            - containerPort: 3000

My server-deployment.yaml is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
  labels:
    app: server-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: server-deployment
  template:
    metadata:
      labels:
        app: server-deployment
    spec:
      containers:
        - name: server
          image: myDocker/k8s-server:latest
          ports:
            - containerPort: 5000

My client-cluster-ip-service.yaml is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
  labels:
    app: server-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: server-deployment
  template:
    metadata:
      labels:
        app: server-deployment
    spec:
      containers:
        - name: server
          image: myDocker/k8s-server:latest
          ports:
            - containerPort: 5000

My server-cluster-ip-service.yaml is:

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    app: server-deployment
  ports:
    - port: 5000
      targetPort: 5000
Davtho1983
  • 3,827
  • 8
  • 54
  • 105
  • 1
    Can you post the YAML of the server deployment? (Also, I assume the client and server are on the same namespace? – Menace Oct 17 '21 at 15:29
  • Amended to include all code :) – Davtho1983 Oct 17 '21 at 15:50
  • Hold on, I am a bit lost, you can query it from another container (busybox), but not from your container, but it is a GKE issue nevertheless? – jabbson Oct 17 '21 at 17:02
  • I had a though, I am not very versed in react, so help me out - is it the client pod that sends this request to the server OR the browser of whoever opens the client pod/app? – jabbson Oct 17 '21 at 17:09
  • I don't know if it is a GKE issue -I'm just out of ideas! I think it should be the client pod that sends the request, but I'm trying to adapt a tutorial that uses websockets in the client pod code instead of fetch. When I run the code locally, the url in the fetch request is "http://localhost:5000", so that does look like it's making the request from the browser – Davtho1983 Oct 17 '21 at 17:24
  • The behaviour I want is the request is made from the client pod to the server pod, via the server service - I don't want the server to be an externally exposed api – Davtho1983 Oct 17 '21 at 17:26
  • The request header in the Network panel show "Referer: http://myIPAddress:3000/" so that looks like it is coming from the pod? – Davtho1983 Oct 17 '21 at 17:36
  • 1
    Out of curiosity, try exec-ing into one of your client pods and running something simple like https://pastebin.com/atAcKN2r – jabbson Oct 17 '21 at 17:52
  • 1
    Also looking at your code (again, I know nothing of express, react, nodejs or javascript), - you print the response object, not the awaited json, what if you try to print json instead (https://pastebin.com/6CfcqnL5). – jabbson Oct 17 '21 at 18:18
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/238251/discussion-between-davtho1983-and-jabbson). – Davtho1983 Oct 17 '21 at 18:20

1 Answers1

1

I was able to see that you and jabbson concluded that probably the issue is with React. Just in case let me share with you that, a common root cause for this kind of issue is that the DNS inside Busybox does not work properly, due to the version (I cannot see in the screenshots and the code, the version that you are using). Most of the cases work with busybox images 1.28.4, empirically talking. You can try using that version.

You can use the following URL’s thread as reference dns can't resolve kubernetes.default and/or cluster.local

  • Thanks - we eventually execed into a client pod, installed node-fetch and executed a script that fetched the service and logged the output - which worked perfectly, so I'm now sure my service is working. I have created a separate question to deal with the React issue - appreciate knowing about busybox it's bound to help in future! – Davtho1983 Oct 18 '21 at 22:16