Having trouble configuring static IPs to PODs attached with MACVLAN interface

Question

Here is the scenario. There is a deployment set through which 2 PODs are created. I am attaching a MACVLAN interface to these PODs for external communication.

Macvlan definition

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: test-macvlandef01
spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "test-macvlandef01",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "datastore": "kubernetes",
        "kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
        "range": "192.168.0.0/24",
        "range_start": "192.168.0.44",
        "range_end": "192.168.0.45"
      }
    }'

Deployment Set

apiVersion: apps/v1
kind: Deployment
metadata:
  name: centos-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: centos
  template:
    metadata:
      labels:
        app: centos
      annotations:
       k8s.v1.cni.cncf.io/networks: "test-macvlandef01"
    spec:
      nodeSelector:
        test: "true"
      containers:
      - name: centos
        image: centos
        imagePullPolicy: IfNotPresent
        command: ["bin/bash", "-c", "sleep 100000" ]
        ports:
        - containerPort: 80

Result. Both PODs have IPs from the allocated pool.

[master1 ~]# kubectl exec -it centos-test-64f8fbf47f-wrjr7  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if61: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether 72:ef:ca:2c:31:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.14.176/32 scope global eth0
       valid_lft forever preferred_lft forever
5: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default 
    link/ether 52:2f:bd:f9:03:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.44/24 brd 192.168.0.255 scope global net1
       valid_lft forever preferred_lft forever
       
[master1 ~]# kubectl exec -it centos-test-64f8fbf47f-vtkst  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ae:e6:4e:95:2a:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.14.175/32 scope global eth0
       valid_lft forever preferred_lft forever
5: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default 
    link/ether 72:fb:b5:90:d0:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.45/24 brd 192.168.0.255 scope global net1
       valid_lft forever preferred_lft forever

Now what I need to configure is, a bigger allocation pool in macvlan definition file, but have only specific 2 IPs to be assigned to the PODs. I tried below configuration.

Macvlan definition

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: test-macvlandef01
spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "test-macvlandef01",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "datastore": "kubernetes",
        "kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
        "range": "192.168.0.0/24",
        "range_start": "192.168.0.40",
        "range_end": "192.168.0.50"
      }
    }'

Deployment Set

apiVersion: apps/v1
kind: Deployment
metadata:
  name: centos-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: centos
  template:
    metadata:
      labels:
        app: centos
      annotations:
       k8s.v1.cni.cncf.io/networks: '[{ "name": "test-macvlandef01","ips": "192.168.0.44"},{"name": "test-macvlandef01","ips": "192.168.0.45"}]'
    spec:
      nodeSelector:
        test: "true"
      containers:
      - name: centos
        image: centos
        imagePullPolicy: IfNotPresent
        command: ["bin/bash", "-c", "sleep 100000" ]
        ports:
        - containerPort: 80

PODs are coming up without MACVLAN interface and also I see no error associated with the POD.

[master1 ~]# kubectl exec -it centos-test-b59db89f7-2vvqx  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether 62:31:fc:64:8f:5b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.14.180/32 scope global eth0
       valid_lft forever preferred_lft forever

[master1 ~]# kubectl exec -it centos-test-b59db89f7-6c75h  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether e6:23:30:ff:bf:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.14.179/32 scope global eth0
       valid_lft forever preferred_lft forever

Please suggest any modifications or additions that would help with the requirement.

Thanks in advance.

Could you please describe your environment in more details? What kind of cluster do you have, cloud or on-premise nodes, Kubernetes/Openshift version, etc. What CNI and what plugins did you install and how exactly? Did you use any guides/manuals to configure MACVLAN? — VAS, Sep 14 '21 at 14:27

score 0 · Answer 1 · answered Aug 06 '21 at 18:25

I want to pay your attention into 2 points below. Partial answer.

From your post I see that you want to use special IP addresses. To use such functionality, according to Extention convention from CNI you may need to use "capabilities": {"ips": true} capability in your Macvlan definition. Something like this:

spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "test-macvlandef01",
      "type": "macvlan",
      "capabilities": {"ips": true}
      "master": "eth0",
      "mode": "bridge",

You can also find good explanation with examples in Attaching a pod to an additional network documentation.

I suppose that you use whereabouts plugin, since "type": "whereabouts" presents in your Macvlan definition. It supports exclusions:

You can also specify ranges to exclude from assignment, so if for example you'd like to assign IP addresses within the range 192.168.2.0/24, you can exclude IP addresses within it by adding them to an exclude list. For example, if you decide to exclude the range 192.168.2.0/28, the first IP address assigned in the range will be 192.168.2.16.

Knowing this fact, you can specify ranges of IPs to exclude from your configuration in accordance with Whereabouts IPAM Config example. Try to add exclude field in Macvlan definition with necessary IPs/subnets, which should be excluded. Possible solution for your particular case:

spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "test-macvlandef01",
      "type": "macvlan",
      "capabilities": {"ips": true}
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.0.0/24",
        "range_start": "192.168.0.40",
        "range_end": "192.168.0.50"
        "exclude": [
           "192.168.0.40/32",
           "192.168.0.41/32",
           ...
        ]
      }
  }'

hi @andrew, i tried with with "capabilities": {"ips": true}" , but got the same result. POD is up but IP is not assigned — Ravi Pillai, Aug 11 '21 at 09:14
Hello @Ravi Pillai. Did you try the second option with Exclude field? — Andrew Skorkin, Aug 12 '21 at 10:45
Hi @Andrew. Excluding is not an option for me as the requirement is to have a macvlan definition with, for example, 30 IPs and use the same for multiple deployments with each deployment having their own set of static IPs. — Ravi Pillai, Aug 13 '21 at 07:09
Ok, @Ravi Pillai. Could you please provide information about your infrastructure - are you using only Macvlan interface or Macvlan and Whereabouts plugin together? I can see information about Whereabouts plugin only from your code, but not in your question. — Andrew Skorkin, Aug 20 '21 at 15:11
Hi @Andrew, I am open to not using whereabouts also. Only need to satisfy the use case. — Ravi Pillai, Aug 30 '21 at 11:14

Having trouble configuring static IPs to PODs attached with MACVLAN interface

1 Answers1