I have a CentOS 7 VM running on VMware that is used for archiving data from various servers, the destination is a FreeNAS MiniXL. Some of this is done over ssh, but others are sent using SMB. In either case rsync is used to compare source/destination. The purpose of using SMB in the first place is to preserve the Windows permissions that are set on FreeNAS. I could be wrong, but I was concerned rsync over SSH would overwrite those (I need to test this.)
The problem is that after a short while of comparing many many small files the SMB client on the CentOS 7 slows down. After a reboot I am able to browse the SMB shares quickly, but not long after some replication jobs start things begin to slow down. For example, ls will take an extremely long time to show any results.
While experiencing the slow down, I tested accessing the shares from other Linux and Windows machines and am able to browse and move files without issue. This leads me to believe it is the SMB client on the CentOS system.
Any thoughts?
CentOS 7 has the SMB destination mounted using vers=3.0 as this has been the most stable for me.
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Linux rhelfs01 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
libsmbclient.x86_64 4.8.3-6.el7_6 @updates
samba.x86_64 4.8.3-6.el7_6 @updates
samba-client.x86_64 4.8.3-6.el7_6 @updates
samba-client-libs.x86_64 4.8.3-6.el7_6 @updates
samba-common.noarch 4.8.3-6.el7_6 @updates
samba-common-libs.x86_64 4.8.3-6.el7_6 @updates
samba-common-tools.x86_64 4.8.3-6.el7_6 @updates
samba-libs.x86_64 4.8.3-6.el7_6 @updates
EDIT :
I have source SMB shares mounted on the same CentOS 7 vm. While browsing the destination is very slow, browsing the source works just fine. After I kill the rsync jobs running on SMB I am able to browse the destination again without slow down.
This looks to be more an issue with rsync over SMB.
rsync.x86_64 3.1.2-6.el7_6.1 @updates
EDIT 2 :
I replicated the probelm on a Windows server using robocopy. Transfer speeds dropped to ~500KBps after a while. However, if I go back to the CentOS 7 server I am able to use DD to write a 10M file to the same destination at ~5.5MBps (Which is normal for my tunnel).
At this point I am thinking it is a problem with samba on the FreeNAS machine. I know very little about the internal workings of samba, but I am going out on a limb and assuming it is multithreaded based on source IP. If I connect from a second IP it has no problem connecting.
Going to be posting my findings to the FreeNAS forums and see what I get. The storage appliance is one of their machines.
EDIT 3 :
Looks like this is more of an issue with the tunnel between the machines being an issue. Even though performance is fine with the other machine, it still transfers files slowly. Opened a thread on Ubiquiti's forums.
