1

kubernetes version: v1.11.0

I run kube-proxy with ipvs mode, got this errors:

7月 03 21:55:08 docker02 kube-proxy[13003]: E0703 21:55:08.316098   13003 ipset.go:156] Failed to make sure ip set: &{{KUBE-NODE-PORT-TCP bitmap:port inet 1024 65536 0-65535 Kubernetes nodeport TCP port for masquerade purpose} map[] 0xc4205e5e40} exist, error: error creating ipset KUBE-NODE-PORT-TCP, error: exit status 2
7月 03 21:55:13 docker02 kube-proxy[13003]: E0703 21:55:13.205413   13003 ipset.go:156] Failed to make sure ip set: &{{KUBE-NODE-PORT-UDP bitmap:port inet 1024 65536 0-65535 Kubernetes nodeport UDP port for masquerade purpose} map[] 0xc4205e5e40} exist, error: error creating ipset KUBE-NODE-PORT-UDP, error: exit status 2
7月 03 21:55:18 docker02 kube-proxy[13003]: E0703 21:55:18.233756   13003 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOAD-BALANCER-LOCAL hash:ip,port inet 1024 65536 0-65535 Kubernetes service load balancer ip + port with externalTrafficPolicy=local} map[] 0xc4205e5e40} exist, error: error creating ipset KUBE-LOAD-BALANCER-LOCAL, error: exit status 2
7月 03 21:55:23 docker02 kube-proxy[13003]: E0703 21:55:23.256248   13003 ipset.go:156] Failed to make sure ip set: &{{KUBE-CLUSTER-IP hash:ip,port inet 1024 65536 0-65535 Kubernetes service cluster ip + port for masquerade purpose} map[] 0xc4205e5e40} exist, error: error creating ipset KUBE-CLUSTER-IP, error: exit status 2
7月 03 21:55:28 docker02 kube-proxy[13003]: E0703 21:55:28.271973   13003 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOAD-BALANCER-SOURCE-CIDR hash:ip,port,net inet 1024 65536 0-65535 Kubernetes service load balancer ip + port + source cidr for packet filter purpose} map[] 0xc4205e5e40} exist, error: error creating ipset KUBE-LOAD-BALANCER-SOURCE-CIDR, error: exit status 2
7月 03 21:55:33 docker02 kube-proxy[13003]: E0703 21:55:33.285863   13003 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOAD-BALANCER-SOURCE-CIDR hash:ip,port,net inet 1024 65536 0-65535 Kubernetes service load balancer ip + port + source cidr for packet filter purpose} map[] 0xc4205e5e40} exist, error: error creating ipset KUBE-LOAD-BALANCER-SOURCE-CIDR, error: exit status 2
7月 03 21:55:36 docker02 kube-proxy[13003]: I0703 21:55:36.485507   13003 proxier.go:701] Stale udp service kube-system/kube-dns:dns -> 10.254.0.2
7月 03 21:55:36 docker02 kube-proxy[13003]: E0703 21:55:36.535070   13003 ipset.go:156] Failed to make sure ip set: &{{KUBE-NODE-PORT-LOCAL-UDP bitmap:port inet 1024 65536 0-65535 Kubernetes nodeport UDP port with externalTrafficPolicy=local} map[] 0xc4205e5e40} exist, error: error creating ipset KUBE-NODE-PORT-LOCAL-UDP, error: exit status 2

According to the source code, I constructed the command manually, e.g: 

sudo ipset create KUBE-LOAD-BALANCE-LOCAL hash:ip,port family inet hashsize 1024 maxelem 65535 -exist

and it's ok，and I can get ipset infos, e.g:

[k8s@docker02 ds]$ sudo ipset list
Name: KUBE-LOAD-BALANCE-LOCAL
Type: hash:ip,port
Revision: 2
Header: family inet hashsize 1024 maxelem 65535
Size in memory: 16528
References: 0
Members:

I have no idea about this problem.

HikoQiu
  • 11
  • 3
  • I'm having the same problem. I have no sets defined and kube-proxy throws errors just like the above. I'm able to actually exec into the proxy container and successfully run the ipset command there. I suspect this is a bug... – Dave McNeill Jul 05 '18 at 17:28
  • @HikoQiu @dave-mcneill Can you tell me your kernel version? Kubernetes actually try to call `ipset` util on that step and exit code `2` means invalid CLI params. – Anton Kostenko Jul 06 '18 at 08:58
  • And OS version too. – Anton Kostenko Jul 06 '18 at 09:04
  • @AntonKostenko uname -a : ` Linux kube-node01 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux `. cat /proc/version: `Linux version 3.10.0-862.2.3.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Wed May 9 18:05:47 UTC 2018` . kube-proxy --version: `Kubernetes v1.11.0` – HikoQiu Jul 06 '18 at 09:50
  • @AntonKostenko Other infos: windows 10 + virtualbox + vagrant + centos/7 (vagrant box)。Actually，I got the same error on my windows10 and Macbook pro virtural machine. – HikoQiu Jul 06 '18 at 10:51
  • After further investigation, I'm pretty sure the problem we are having is directly related to https://bugzilla.redhat.com/show_bug.cgi?id=1496859. strace shows ipset running with the 'comment' param, which is not (yet) supported on RHEL/CentOS 7 ([pid 25596] write(2, "Unknown argument: `comment'", 27 ) – Dave McNeill Jul 06 '18 at 12:00
  • @DaveMcNeill you're right, I found the issue on github just now: https://github.com/kubernetes/kubernetes/issues/65611 – HikoQiu Jul 06 '18 at 13:53

1 Answers1

-1

After starting the kube-proxy in ipvs mode, appropriate ipset entries will be created automatically.

Looks like ipset with the KUBE-LOAD-BALANCE-LOCAL name already exists in the system when you're trying to run kube-proxy in ipvs mode.

Try to delete KUBE-LOAD-BALANCE-LOCAL ipset entry and after that run kube-proxy in ipvs mode.

To delete ipset entry you can use the below command:

ipset destroy KUBE-LOAD-BALANCE-LOCAL

Update:

@DaveMcNeill is right. This is a known bug in RedHat/CentOS systems.

It has been fixed in Kubernetes by the below commit:

fix ipset creation fails on centos. issue 65461

In this case, you should wait for the release with this commit included, or use another OS (Debian for example).

Akar
  • 574
  • 3
  • 8