Cannot update Calico CNI config on Kubernetes cluster

Question

I discovered that, until a few months ago, the "hostPort" configuration for Pods was not going to work with CNI based integrations. This meant that, for any Kubernetes cluster using Calico, it was not possible to directly expose a Pod's port directly on a certain Node's port, without using a Service or flagging hostNetwork=true (which is a little bit extreme).

Starting from Kubernetes 1.7.0 it is possible but it is necessary to change Calico configuration in order to let the new "portmap" CNI plugin in, which is what I'm trying to do, without success. I am starting from a new IBM Bluemix Container Service cluster.

My calico-node DaemonSet has the following CNI_NETWORK_CONFIG environmental variable:

{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "type": "calico",
  "etcd_endpoints": "__ETCD_ENDPOINTS__",
  "etcd_key_file": "__ETCD_KEY_FILE__",
  "etcd_cert_file": "__ETCD_CERT_FILE__",
  "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
  "log_level": "info",
  "mtu": 1480,
  "ipam": {
    "type": "calico-ipam"
  },
  "policy": {
    "type": "k8s",
    "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
    "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
  },
  "kubernetes": {
    "kubeconfig": "__KUBECONFIG_FILEPATH__"
  }
}

What I did here was just trying to replace it with the following configuration:

{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins": [{
    "type": "calico",
    "etcd_endpoints": "__ETCD_ENDPOINTS__",
    "etcd_key_file": "__ETCD_KEY_FILE__",
    "etcd_cert_file": "__ETCD_CERT_FILE__",
    "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
    "log_level": "info",
    "mtu": 1480,
    "ipam": {
      "type": "calico-ipam"
    },
    "policy": {
      "type": "k8s",
      "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
      "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
    },
    "kubernetes": {
      "kubeconfig": "__KUBECONFIG_FILEPATH__"
    }
  },
    {
      "type": "portmap",
      "snat": true,
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

calico-node pods were running successfully after a forced reboot, but my own Pods keep getting stuck in "Pending" status during initialization, with the event "Error syncing pod" from "kubelet NODE_IP".

I'd appreciate some help on this issue. Thanks in advance.

score 1 · Answer 1 · answered Nov 06 '17 at 15:15

1

What you have looks reasonable as far as the contents, I think the problem may be that you need to change the name of the config file from ending in .conf to .conflist. There is a PR up with some WIP changes https://github.com/projectcalico/calico/pull/903 for enabling hostport in the calico manifests, you can compare it with what you have done.

If you set the filename through the daemonset you should remove the previous config file on the hosts because the released install-cni container does not clean up the previous config and I am not sure which config file the kubelet would use.

answered Nov 06 '17 at 15:15

Erik Stidham

201
1
4

I've changed the filename from `.conf`to `.conflist`and now, at least, `calico-node` runs correctly and so the other pods. Unfortunately I still cannot access the node's port. Maybe it's the old config file making a conflict or, more likely, a firewall rule somewhere in the stack. Still investigating on this part but thank you. – Emanuele Casadio Nov 07 '17 at 08:49
Ok, I have added a plain new Node into the cluster and I still cannot access the port, so it's very likely a bad firewall configuration or something like that. I still cannot figure out where the exact problem is. – Emanuele Casadio Nov 07 '17 at 09:30
Since you're not sure what is happening with your traffic, you could use tcpdump and `sudo iptables-save -c | grep DROP` (to see if the traffic is arriving to the host and if it is being dropped by the host). – Erik Stidham Nov 20 '17 at 15:37
Turns out IBM has a problem with their cloud so that this configuration just cannot work until they fix it. :( – Emanuele Casadio Dec 19 '17 at 16:11

Cannot update Calico CNI config on Kubernetes cluster

1 Answers1