Why does l2arc take space in RAM rather than on the l2arc device itself?

[unknown-user-9246102101]

Why does l2arc take space in RAM rather than on the l2arc device itself? Does it lose cache effectiveness by doing so? Surely given the cache heirarchy it would still be better than spinning disk, right?

[IvanVolosyuk]

ZFS has to know if the object exists in L2ARC to read it from there. It will be too expensive to read the L2ARC device just to understand if the object is there. Thus, L2ARC uses some ARC RAM to store the dictionary of what's available in the l2ARC device and where exactly. Having too large L2ARC devices may cause large portion of ARC to be used by the dictionary and be unavailable for ARC (first layer cache). This may actually slow the system down. Different workloads may need different sizes of L2ARC. When unsure don't add L2ARC larger than 5x of the system ram.

…

On Sat, Aug 14, 2021 at 5:35 AM __blank__ ***@***.***> wrote: Why does l2arc take space in RAM rather than on the l2arc device itself? Does it lose cache effectiveness by doing so? Surely given the cache heirarchy it would still be better than spinning disk, right? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#12479>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HN6L4U55Y3J55S5K6LT4VXXLANCNFSM5CEFN77Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[grahamperrin]

From OpenZFS: All about the cache vdev or L2ARC | Klara Inc. (undated):

… exactly how much RAM is necessary to index the L2ARC, …