Running Graylog in a Kubernetes cluster behind an ingress controller

Question

I'm trying to set up Graylog within my Kubernetes cluster as described here. The problem I'm running into is the definition of the environment variable GRAYLOG_HTTP_EXTERNAL_URI. The documentation tells me to enter "my IP adress" and from what I was able to find out, the variable is ment to tell the browser where to find the Graylog API.

But my cluster is accessed through a NGINX reverse proxy serving as an ingress controller, which means the browser can't access the Graylog pod directly and even less through http, so it don't really know what value I should assing there. I tried the public IP address of the ingress controller but all I'm getting is a 503. Is there a way to allow access to the Graylog API while still keeping the service protected behind the ingress controller?

Mark Watney · Accepted Answer · 2020-03-24T14:09:39.113

It really depends on how are you exposing it. By default it's not exposed to the outside world. We have graylog3 service of type NodePort, so we have only an internal IP that can be accessed from another pod or used to expose it with ingress.

$ kubectl get service -o wide
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE   SELECTOR
es6          NodePort    10.23.244.28    <none>        9200:30001/TCP,9300:30002/TCP    54m   service=es-deploy
graylog3     NodePort    10.23.242.128   <none>        9000:30003/TCP,12201:30004/TCP   54m   service=graylog-deploy
kubernetes   ClusterIP   10.23.240.1     <none>        443/TCP                          57m   <none>
mongo        ClusterIP   10.23.243.160   <none>        27017/TCP                        54m   service=mongo-deploy

If we curl this service and port from another pod we have the following output:

$ kubectl exec -ti ubuntu -- curl 10.23.242.128:9000
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="robots" content="noindex, nofollow">
    <meta charset="UTF-8">
    <title>Graylog Web Interface</title>
    <link rel="shortcut icon" href="http://your_ip_address:30003/assets/favicon.png">

  </head>
  <body>
    <script src="http://your_ip_address:30003/config.js"></script>

    <script src="http://your_ip_address:30003/assets/vendor.4024e2a8db732781a971.js"></script>

    <script src="http://your_ip_address:30003/assets/polyfill.a5e2fb591e8fd54ee4ef.js"></script>

    <script src="http://your_ip_address:30003/assets/builtins.a5e2fb591e8fd54ee4ef.js"></script>

    <script src="http://your_ip_address:30003/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.b864ba54b438ac0bdc48.js"></script>

    <script src="http://your_ip_address:30003/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.bcc87290018e859a8a9e.js"></script>

    <script src="http://your_ip_address:30003/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.8ae7cb13983ce33eeb5b.js"></script>

    <script src="http://your_ip_address:30003/assets/app.a5e2fb591e8fd54ee4ef.js"></script>

  </body>
</html>

As can be seen, there is a reference for http://your_ip_address:30003. If we leave it this way, the application will break because it's referencing something nonexistent.

So I'll change 2 things, make it visible from the outside world with an ingress and change GRAYLOG_HTTP_EXTERNAL_URI to the correct IP I'll get:

1 - Creating ingress rule to expose Graylog:

This is how my ingress manifest is looking

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: graylog
spec:
  backend:
    serviceName: graylog3
    servicePort: 9000

$ kubectl get ingresses 
NAME      HOSTS   ADDRESS          PORTS   AGE
graylog   *       34.107.139.231   80      56s

2 - Edit our GRAYLOG_HTTP_EXTERNAL_URI and substitute http://your_ip_address:30003 for http://34.107.139.231:80.

Observe that here I'm changing port from 30003 to 80 since our ingress rule is exposing on port 80.

$ kubectl edit deployments graylog-deploy

Changes made, now let's curl this port from any console (give it some time for the pods to be recreated):

$ curl 34.107.139.231:80
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="robots" content="noindex, nofollow">
    <meta charset="UTF-8">
    <title>Graylog Web Interface</title>
    <link rel="shortcut icon" href="http://34.107.139.231:80/assets/favicon.png">

  </head>
  <body>
    <script src="http://34.107.139.231:80/config.js"></script>

    <script src="http://34.107.139.231:80/assets/vendor.4024e2a8db732781a971.js"></script>

    <script src="http://34.107.139.231:80/assets/polyfill.a5e2fb591e8fd54ee4ef.js"></script>

    <script src="http://34.107.139.231:80/assets/builtins.a5e2fb591e8fd54ee4ef.js"></script>

    <script src="http://34.107.139.231:80/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.b864ba54b438ac0bdc48.js"></script>

    <script src="http://34.107.139.231:80/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.bcc87290018e859a8a9e.js"></script>

    <script src="http://34.107.139.231:80/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.8ae7cb13983ce33eeb5b.js"></script>

    <script src="http://34.107.139.231:80/assets/app.a5e2fb591e8fd54ee4ef.js"></script>

  </body>
</html>

Now we can see http://34.107.139.231:80/ as expected and the page can be loaded perfectly.

If you have a domain name that's redirecting to your application IP, put it in this variable.

But isn't this undermining the purpose of having an ingress controller, which is to prevent direct access to your resources, posing a security issue? — TigersEye120, Mar 24 '20 at 13:20
My example is trying to show you how to understand which IP should be used in `GRAYLOG_HTTP_EXTERNAL_URI`. You are right, using ingress is a much better approach but for my example I choose to expose it using `LoadBalancer` just to make things faster and more didactic. In the end, doesn't matter how are you exposing it, `GRAYLOG_HTTP_EXTERNAL_URI` should be the external ip/name used to access Graylog dashboard. — Mark Watney, Mar 24 '20 at 13:24
I just realised that I might not have formulated my question clearly: Accessing the API while still keeping the service protected behind the ingress is the actual point of what I'm trying to do here, so exposing the entire thing as a load balancer is unfortunately not an option for me. — TigersEye120, Mar 24 '20 at 13:32
I edited my answer changing the approach. Now we are exposing it using `ingress` instead of `LoadBalancer`. Please check it and let me know if it helped you. — Mark Watney, Mar 24 '20 at 14:10
Strange, that's basically the same thing I'm doing. I'll look more into this tomorrow, maybe I'll find a mistake I made. Anyway, thanks for your help. :D — TigersEye120, Mar 24 '20 at 14:21
If you need more help, please let me know. Posting your custom manifest and I can try to reproduce here. — Mark Watney, Mar 24 '20 at 14:22
Alright that problem seems to be solved, I discovered some copy-paste errors in the elasticsearch yaml and I also used port 30003 with the ingress instead of 9000. Now I still need to teach it to use https but that's a different story. Thanks again for your time. — TigersEye120, Mar 25 '20 at 10:05

Running Graylog in a Kubernetes cluster behind an ingress controller

1 Answers1