0

I'm trying to setup Redis cluster in Kubernetes. The major requirement is that all of nodes from Redis cluster have to be available from outside of Kubernetes. So clients can connect every node directly. But I got no idea how to configure service that way.

Basic config of cluster right now. It's ok for services into k8s but no full access from outside.

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: redis-cluster
      labels:
        app: redis-cluster
    data:
      redis.conf: |+
        cluster-enabled yes
        cluster-require-full-coverage no
        cluster-node-timeout 15000
        cluster-config-file /data/nodes.conf
        cluster-migration-barrier 1
        appendonly no
        protected-mode no
    ---
    apiVersion: v1
    kind: Service
    metadata:
      annotations:
        service.alpha.kubernetes.io/tolerate-unready-endpoints: "false"
      name: redis-cluster
      labels:
        app: redis-cluster
    spec:
      type: NodePort
      ports:
      - port: 6379
        targetPort: 6379
        name: client
      - port: 16379
        targetPort: 16379
        name: gossip
      selector:
        app: redis-cluster
    ---
    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: redis-cluster
      labels:
        app: redis-cluster
    spec:
      serviceName: redis-cluster
      replicas: 6
      template:
        metadata:
          labels:
            app: redis-cluster
        spec:
          hostNetwork: true
          containers:
          - name: redis-cluster
            image: redis:4.0.10
            ports:
            - containerPort: 6379
              name: client
            - containerPort: 16379
              name: gossip
            command: ["redis-server"]
            args: ["/conf/redis.conf"]
            readinessProbe:
              exec:
                command:
                - sh
                - -c
                - "redis-cli -h $(hostname) ping"
              initialDelaySeconds: 15
              timeoutSeconds: 5
            livenessProbe:
              exec:
                command:
                - sh
                - -c
                - "redis-cli -h $(hostname) ping"
              initialDelaySeconds: 20
              periodSeconds: 3
            volumeMounts:
            - name: conf
              mountPath: /conf
              readOnly: false
          volumes:
          - name: conf
            configMap:
              name: redis-cluster
              items: 
              - key: redis.conf
                path: redis.conf
Shtlzut
  • 2,174
  • 1
  • 16
  • 14
  • Do you know how many Redis nodes there will be, or does the solution have to be dynamic? – ewramner Jul 06 '18 at 14:28
  • @ewramner dynamic, but for a test case, there will be enough fixed amount -6 or 8 – Shtlzut Jul 06 '18 at 14:55
  • If it is static you can define one additional service per pod as the pods get stable names (-0, -1, ...). Those services can have stable external IPs. That should work, but if you need to make it dynamic you will have to script it. – ewramner Jul 07 '18 at 06:55

1 Answers1

1

Given:

    spec:
      hostNetwork: true
      containers:
      - name: redis-cluster
        ports:
        - containerPort: 6379
          name: client

It appears that your StatefulSet is misconfigured, since if hostNetwork is true, you have to provide hostPort, and that value should match containerPort, according to the PodSpec docs:

hostPort integer - Number of port to expose on the host. If specified, this must be a valid port number, 0 < x < 65536. If HostNetwork is specified, this must match ContainerPort.

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#containerport-v1-core

mdaniel
  • 31,240
  • 5
  • 55
  • 58