When NFS Client try to create a file on shared folder, operation not permitted occurs, both server and client are in containers

Question

I've googled a lot and found some similar cases. However, none of them works in my case. Not sure if it has something to do with Mac.

I will try to describe my question as clearly as possible.

• I use KIND to run a local Kubernetes cluster with 3 nodes. I mount a volume in the master node /data/k8s from my local MacOS /tmp/k8s
• I create a NFS Server on the master node, export /data/k8s
• I also install NFS on the slave node as NFS Client, and mount master node /data/k8s to my the slave node local /data/k8s

and here is the problem:

• Everything works well on master node, no problem.
• On slave node, I CANNOT create a regular file while I CAN create a folder
• On slave node, I CAN revise any files created on master node
• Any files created on master node would sync without problems on slave node.

and here is some information:

• The OS (Both master and slave are identical)

NAME="Ubuntu"
VERSION="19.10 (Eoan Ermine)"
ID=ubuntu
ID_LIKE=debian

• mount information on master node (The space is initially mounted from host MacOS)

root@test2-control-plane:/data/k8s# mount | grep /data
grpcfuse on /data/k8s type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

• exports setting on master node

/data/k8s/ *(fsid=0,rw,sync,no_root_squash)

• etab information on master node

*(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,fsid=0,anonuid=65534,anongid=65534,sec=sys,rw,secure,no_root_squash,no_all_squash)

• permission information on master node

root@test2-control-plane:/data/k8s# ls -al
total 24
drwxrwxrwx 7 root root  224 Dec 25 01:50 .
drwxrwxrwx 3 root root 4096 Dec 24 12:33 ..

• mount information on slave node

root@test2-worker:/data/k8s# mount | grep /data
172.18.0.4:/data/k8s on /data/k8s type nfs (rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.18.0.4,mountvers=3,mountport=45257,mountproto=udp,local_lock=all,addr=172.18.0.4)

• permission information on slave node

root@test2-worker:/data/k8s# ls -al
total 24
drwxrwxrwx 8 root root  256 Dec 25 02:17 .
drwxrwxrwx 3 root root 4096 Dec 24 13:23 ..

• What I've tried:
  • I've checked SELinux

  root@test2-control-plane:/data/k8s# sestatus
  SELinux status:                 disabled
  

  • I've tried to add all_squash and add anonuid and anongid with a specific user, and I define the user with identical user_id and group_id on both master node and slave node. And then switch to the defined user on my slave node.
  • Have tried so many that I can't remember them all.

I know it may be a specific case. If anyone could help will be so so much appreciated.