2

My setup has 256 GB of RAM, of which 200 GB are dedicated to arc, and 2 SSDs for l2arc. It's being up since several days and steadily serving files to many clients, yet it seems that the l2arc is still not warmed up properly. The cache report of zpool iostat -v says:

cache           -      -      -      -      -      -
  sdy       4.84G   889G      0      0    203    819
  sdz       4.84G   889G      0      0    202    826

I'm seeing a miss ratio for arc constantly around 10%:

$ arcstat 2
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
09:18:03    78     7      9     0    0     7  100     0    0  200G  199G    31G
09:18:05   21K  1.0K      8     5    1  1.0K   50     5    1  200G  199G    31G
09:18:07   18K  1.0K      8    40    1  1.0K   47     2    1  199G  199G    31G
09:18:09   18K  1.0K      7    44    1  1.0K   48     3    1  200G  199G    31G

so I'd say that this should lead to a population of the l2arc, which instead seems to not happen with the two cache SSDs stuck at an occupation of about 4.8 GB since days. As a consistency check the l2arc miss ratio is constantly 100%:

$ arcstat -f l2bytes,l2read,l2miss,l2miss% 2
l2bytes  l2read  l2miss  l2miss%
      0       6       6      100
      0    1.0K    1.0K      100
      0    1.0K    1.0K      100
      0    1.0K    1.0K      100

For sure I'm either missing something or having a faulty setup, so I'd need some guidance.

Nicola Mori
  • 281
  • 1
  • 7
  • Did you solve the problem in the meantime? – Binarus Oct 13 '22 at 09:58
  • No, but maybe I understood why. My use case is mainly the processing of very large (several GB) data files. These are processed in parallel by several computing nodes during batch processing. During a batch each file is accessed once by a single computing process, so it is transferred to ARC but soon purged since other big files are being accessed. Thus the file is purged from the ARC before it reaches the minimum number of accesses for becoming eligible for L2ARC. At least, this is what I understood might be happening according to my (admittedly poor) knowledge of ZFS internals. – Nicola Mori Oct 14 '22 at 10:08

1 Answers1

1

L2ARC is populated only from soon-to-be-evicted ARC buffers. If ARC churn rate is too low, or too high, L2ARC can have issue warming up.

To increase L2ARC utilization, you can try to set l2arc_noprefetch=0 and, if you are using ZFS 2.0.x, l2arc_headroom=0 (if using an older ZFS version, you need to specify a value as l2arc_headroom=32).

shodanshok
  • 47,711
  • 7
  • 111
  • 180
  • I set both l2arc_noprefetch and l2arc_headroom to 0 but after one day the l2arc size is still 4.84 GB per disk and I still get 100% l2arc miss rate. – Nicola Mori Feb 10 '22 at 08:19