0

I'm a newbie on K8s, please forgive me if this is a straightforward question.

TL;DR

I deployed a classic server-client architecture using a k8s service to reach server by using service's FQDN.

When I apply only the below yaml, server is reachable from client (i.e.: wget -O- my-service returns Apache hello page).
When I apply also kube-prometheus as in their quickstart (kubectl apply --server-side -f manifests/setup -f manifests) I can no more reach server (wget returns Connection Refused).

I'm using kind on Windows 10

The problem

I need to deploy Prometheus on kubernetes to scale pods based on some custom metrics. My pods follows the classic server-client architecture; the below yaml is something similar to what I have actually deployed on k8s.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache
  labels:
    app: apache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apache
  template:
    metadata:
      labels:
        app: apache
    spec:
      containers:
      - name: apache
        image: httpd:2.4-alpine
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: alpine
  labels:
    app: alpine
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alpine
  template:
    metadata:
      labels:
        app: alpine
    spec:
      containers:
      - name: alpine
        image: alpine:3.15
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
  labels:
    service: my-service
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: server-tcp
  selector:
    app: apache

If I deploy this yaml before following kube-prometheus quickstart, apache is reachable from alpine;
i.e. (from alpine):

$ wget -q -O- my-service
<html><body><h1>It works!</h1></body></html>

If I deploy it after deploying kube-prometheus, apache is no more reachable from alpine;
i.e. (from alpine):

$ wget -q -O- my-service
Connecting to my-service
wget: can't connect to remote host: Connection refused

I don't know whether this is a bug from kube-prometheus or I'm doing something wrong. I tried to google a lot but I couldn't find any working solution for this problem.
I tried with iptables --policy FORWARD ALLOW on apache pod, but it didn't work (no permission even if command was issued as root).

Any help is appreciated.

I'm using kind on Windows 10

Update

alpine pod has no logs.
Here are the logs for apache pod (with IPs and PID removed)

AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using pod_internal_ip. Set the 'ServerName' directive globally to suppress this message
[Tue Feb 01 12:30:58.430151 2022] [mpm_event:notice] AH00489: Apache/2.4.52 (Unix) configured -- resuming normal operations
[Tue Feb 01 12:30:58.430280 2022] [core:notice] AH00094: Command line: 'httpd -D FOREGROUND'
Matteo Meil
  • 1,192
  • 10
  • 20
  • There are many manifest files from prometheus to check in order to help you. It would help if you see and post the logs of the pods created using `kubectl logs` – Felipe Feb 01 '22 at 07:29
  • `alpine` has no logs while `apache` shows normal startup logs. See the update for details – Matteo Meil Feb 01 '22 at 12:33
  • after you deploy `prometheus`, can you exec into `apache` and do a `curl http://localhost` just to make sure the server is still listening? – pkaramol Feb 01 '22 at 17:40
  • Yes, it's responding with hello page, i.e.: `wget -q -O- localhost` from inside `apache` container outputs `

    It works!

    `     – Matteo Meil Feb 01 '22 at 20:04

0 Answers0