zfs send to NAS with PIGZ compression slowness

Question

The following command that I use to backup used to take 5 minutes to complete and since last week it is taking around 25 minutes. This is after restarting a switch stack the server is connected to. The NAS 192.6.222.116 is in a different subnet to the server. How should I go about troubleshooting this issue?

zfs send -Rv shares1-pool/homedir@Cron-Snap-OraZ-Mon-030723 | pigz > /NAS/test/archive_shares1.gz

The server is not struggling for memory,

root@mars:~# echo ::memstat|mdb -k
Page Summary                            Pages             Bytes  %Tot
---------------------------- ----------------  ----------------  ----
Kernel                                 724362              5.5G    9%
Defdump prealloc                       173879              1.3G    2%
ZFS                                    912162              6.9G   11%
Anon                                  1282098              9.7G   16%
Exec and libs                           44767            349.7M    1%
Page cache                             986194              7.5G   12%
OSM                                   1792000             13.6G   22%
Free (cachelist)                       469418              3.5G    6%
Free (freelist)                       1774352             13.5G   22%
Total                                 8159232             62.2G

iostat,

         cpu
     us sy st id
     13  4  0 84

                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.4    0.0    0.8  0.0  0.0    0.0    0.1   0   0 sd3
    0.2    0.4    0.8    0.8  0.0  0.0    0.0    3.2   0   0 sd5
    0.4   48.4   51.2 1507.4  0.0  0.1    0.0    1.5   0   1 ssd11
    0.0   49.2    0.0 1507.4  0.0  0.1    0.0    1.6   0   1 ssd12
    0.0   23.4    0.0  729.6  0.0  0.0    0.0    1.3   0   1 ssd13
    0.0   23.0    0.0  729.6  0.0  0.0    0.0    1.1   0   1 ssd14
   50.8   15.8 5527.4  108.8  0.0  0.3    0.0    4.3   0  24 ssd15
   53.4   16.2 4624.1  108.8  0.0  0.3    0.0    4.7   0  24 ssd16
    0.0   15.6    0.0  177.8  0.0  0.0    0.0    0.9   0   0 ssd19
    0.0   15.8    0.0  177.8  0.0  0.0    0.0    0.9   0   0 ssd20
   39.2   58.8  320.6 1293.5  0.0  0.2    0.0    1.8   0   6 ssd21
   48.8   57.0  334.9 1293.5  0.0  0.2    0.0    2.2   0   7 ssd22
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 st2
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 st3
    0.0  305.2    0.0 9766.4  7.0 10.0   22.9   32.8 100 100 192.6.222.116:/nfs_share
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 192.6.223.116:/nfs_share
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 192.6.224.40:/nfs_share

dlstat

  LINK    IPKTS   RBYTES    OPKTS   OBYTES
   net0   50.82G    9.10T   39.06G  134.91T

vmstat

 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s3 s5 sd sd   in   sy   cs us sy id
 0 0 0 32028280 10823336 589 877 3469 0 0 0 0 7 7 40 40 13788 14114 9252 6 2 91

mpstat

 CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  st idl
   0   72  10  439  1053  370  920   55   60  114    0  1207    6   2   0  93
   1  159  18 1190  2957 2276 1362   42   87  130    0  2185    7   3   0  90
   2  110  19  512  1045  287 1401   56   86  135    0  2209    7   2   0  91
   3   97  10  370   571   45  946   53   61  116    0  1362    6   2   0  92
   4   73   9  450   929  437  904   52   59  115    0  1201    6   2   0  93
   5  158  19  536  2706 1933 1411   50   88  134    0  2375    7   3   0  90
   6  110  16 1137  2476 1821 1349   57   84  132    0  2236    7   3   0  90
   7   97   9  373  2050 1518  959   54   61  115    0  1339    6   2   0  92

On the NAS, there isn't any errors/warnings and seems to have enough space available for the backup.

Thanks in advance.

What kind of performance do you get if you send it to another NAS system? Or to a local file? — Andrew Henle, Jul 04 '23 at 17:24
I would check the bandwidth between the source server and your NAS with something as `iperf` — shodanshok, Jul 05 '23 at 08:07
@AndrewHenle I get the same slow performance with a different NAS on a different subnet. When I do it locally, it is performing as fast as it used to do with the NAS. — najarajar, Jul 06 '23 at 08:11

zfs send to NAS with PIGZ compression slowness

0 Answers0