1

I installed MicroK8s on two Ubuntu 20.04 systems. One is assigned as the CP and I joined the second system as a node to the cluster.

I am trying to spin up a couple of pods with syslog-ng to consume syslog traffic on port UDP/514. I am generating some synthetic syslog traffic from an external system to see the load balancing in action. However, I only see one of the pods consuming data at any given time. Following is my yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: syslogng-server
  labels:
    app: syslogng-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: syslogng-server
  template:
    metadata:
      labels:
        app: syslogng-server
    spec:
      containers:
        - name: syslogng-server
          image: 'balabit/syslog-ng:3.25.1'
          resources:
            requests:
              cpu: 10m
          ports:
            - containerPort: 514
              name: udp
              protocol: UDP
          volumeMounts:
            - name: config-volume
              mountPath: /etc/syslog-ng/conf.d
      volumes:
        - name: config-volume
          configMap:
            name: syslogng-d

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: syslogng-d
  labels:
    app: syslogng-server
data:
  syslogng-test.conf: |
    ##################################################
    options {
            create_dirs(yes);
            owner(root);
            group(root);
            perm(0640);
            dir_owner(root);
            dir_group(root);
            dir_perm(0750);
    };


    ##################################################
    source s_net {
                   tcp(ip(0.0.0.0) port(514));
                   udp(ip(0.0.0.0) port(514));
    };

    ##################################################
    destination d_host-specific {
            file("/var/log/firewalls.log");
    };

    log {
            source(s_net);
            destination(d_host-specific);
    };

---

apiVersion: v1
kind: Service
metadata:
  name: syslogng-server-service
  labels:
    app: syslogng-server-service
spec:
  selector:
    app: syslogng-server
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  # loadBalancerIP is optional. MetalLB will automatically allocate an IP
  # from its pool if not specified. You can also specify one manually.
  # loadBalancerIP: x.y.z.a
  ports:
    - name: udp
      protocol: UDP
      port: 514
      targetPort: 514

I was hoping to see both pods actively consuming the syslog data but that is not the case. Am I doing something wrong? Thank you for the assistance.

Palezvar
  • 15
  • 2

2 Answers2

0

With MetalLB in layer2 mode, one node in the cluster takes "ownership" of the service, all traffic for the LoadBalancer IP goes to that node and it then hands over to kube-proxy to get the traffic to the Service / Pods.

It wouldn't be too surprising for traffic not to be balanced equally, especially if there is only one source. There could be a number of reasons for that, such as kube-proxy preferring to route traffic to the Pod on the same Node or some sort of "source IP affinity" happening.

TL;DR yes! MetalLB isn't the best at load balancing in Layer 2 mode.

There's more information in the MetalLB Docs.

OllyTheNinja
  • 566
  • 3
  • 16
0

Currently, I am facing the same problem, It happened in the UDP situation. Because of using iptables as a load balancer, it is just a DNAT with a statistic module to load balance the traffic.

And the kernel has some cache called conntrack in the kernel. The conntrack will store the mapping of src port and src IP to DST port and DST IP.

In TCP's situation, it is a stateful protocol, the conntrack's cache would be clear after the connection is terminated or just TTL counts down to 0.

However, in UDP's situation, it is a stateless protocol, how does the conntrack cache work? It just keeps the mapping with particular TTL, if the TTL counts to 0, the record would be removed from the conntrack cache.

And at my environment, my UDP TTL is 2 minutes, it will refresh the TTL when the packet belongs to this stream, so in your situation, if there are 2 minutes your application doesn't generate any log(send UDP traffic), it may load balancing to another Syslog pod by running DNAT chain's rule.

you can check your conntrack cache with conntrack -L or iptstate to check the UDP TTL and survey the iptables rule with iptabless-save.

Jerry Drung
  • 1