It is not clear to me what you're complaining about.
If Ethernet, build a virtual RAM disk on both client and server and rsync from and to it. Also ensure that you are not tunneling through ssh nor rsync has compression options enabled, otherwise CPU utilization on client or server might have an impact on transfer speed.
Doing this will check your ethernet and if there are no issues the bandwidth should not drop/pike but will be constant.
After that, benchmark your disks. Both client and server. Probably they are affecting your transfer speed.
You did not provide enough informations to go any further, I just want to point you in the right direction: exclude it's a matter of network, then go check client's io read speed and server's io write speed on all destination disks.
I bet the bottleneck it's the destination disks, especially if you run software raid or if they are standalone desktop sata HDDs.