add partition on ssd/nvme as L2-arc read-cache for hdd pool

[n0xena]

As this seems a rare use case I wasn't able to find conclusive informations, hence asking here:
I use a 8x 3TB hdd raidz2 pool connected via a pci-e 3.0 x1 s-ata hba (works well - sequential reads can sustain the full bandwidth). As I also have my games stored on it from time to time I get some loading hick-ups when loading smsll random assets from big archives (as an example: gta v). As I only have 32gb of ram the primary ram arc can't hold enough assets in hot ram for smooth gameplay.
I have only a 500gb system drive (nvme, pci-e 4.0 x4) and about 80gb-120gb to spare on it.
My question: Do I understand L2 arc correctly as additional cache tier? Would it make sense to allocate a 100gb on the nvme as L2 arc read-cache so that on load most of the game can loaded from the hdds onto tge flash and from there way faster into ram?
Or am I missing something important?
For testing I used the s-ata ssd from my laptop to confirm the loading hick-ups are caused by the bad random access performance of the pool - when playing from the ssd there're none - but I don't want to waste my laptops ssd.
Why not put all my games on the nvme: a) it would make the pool obsolete but b) due to thier size they just won't fit. Instead of moving the games aroubd I would rather try to leverage thw zfs L2 arc as read cache.

Does my idea makes any sense - or am I miles away due to misunderstanding basics?
I don't need a ZIL as I don't have the workload for it - but the pool does get some nice sequential writes over 200mb/s sustained.

[IvanVolosyuk]

I use all 3 variants: - Some games are on hdd raidz pool - Some games are on hdd raidz pool with l2arc - Some games are on non-redundant single disk ssd pool I regularly backup to raidz hdd pool. Usually, l2arc helps to get better load times and helps quite a bit and I use it for most of my games. It doesn't fully eliminate cache misses and certain games are very sensitive to disk latency. In Elden Ring I can get half a second lags (frame drops) in some areas when running in this configuration, thus I use an ssd only pool for those.

…

On Sat, Jul 13, 2024 at 1:23 AM n0xena ***@***.***> wrote: As this seems a rare use case I wasn't able to find conclusive informations, hence asking here: I use a 8x 3TB hdd raidz2 pool connected via a pci-e 3.0 x1 s-ata hba (works well - sequential reads can sustain the full bandwidth). As I also have my games stored on it from time to time I get some loading hick-ups when loading smsll random assets from big archives (as an example: gta v). As I only have 32gb of ram the primary ram arc can't hold enough assets in hot ram for smooth gameplay. I have only a 500gb system drive (nvme, pci-e 4.0 x4) and about 80gb-120gb to spare on it. My question: Do I understand L2 arc correctly as additional cache tier? Would it make sense to allocate a 100gb on the nvme as L2 arc read-cache so that on load most of the game can loaded from the hdds onto tge flash and from there way faster into ram? Or am I missing something important? For testing I used the s-ata ssd from my laptop to confirm the loading hick-ups are caused by the bad random access performance of the pool - when playing from the ssd there're none - but I don't want to waste my laptops ssd. Why not put all my games on the nvme: a) it would make the pool obsolete but b) due to thier size they just won't fit. Instead of moving the games aroubd I would rather try to leverage thw zfs L2 arc as read cache. Does my idea makes any sense - or am I miles away due to misunderstanding basics? I don't need a ZIL as I don't have the workload for it - but the pool does get some nice sequential writes over 200mb/s sustained. — Reply to this email directly, view it on GitHub <#16344>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HJXEUEM3OI4UR5Y7FTZL7YHDAVCNFSM6AAAAABKZEC6WSVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWHEZDONJSGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[n0xena]

Does the l2arc help for those games stored on that pool?

[IvanVolosyuk]

I think it does, though it is hard to measure it objectively. I can look at the hit rate and average latency of disk access using 'zpool iostat -lv'. I can see large number of read requests goes to l2arc and overall latency is around 1ms, while raidz average ~15ms and l2arch latency is ~150us.

[n0xena]

Nice numbers.
What are the commands to check hits/misses so I can try to evaluate if it will help in my use case?
How should I start to profile the use of the l2arc?

[IvanVolosyuk]

• I usually use zpool status -lv for average latency, you can get latency distribution as well per block device and vdev using zpool iostat -wv.
• arc_summary gives percent of hits and misses as well as amount of data read / written to l2arc in 'L2ARC breakdown' and 'L2ARC I/O' sections.
• arcstat -f time,read,hits,l2hits,l2miss,l2hit% 60 can give some more details about arc, I don't use it much.
• You can find more counters is proc in Linux, e.g. /proc/spl/kstat/zfs/arcstats

There are several tunables which allows to adjust fill rate of l2arc. You can find them here: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html

[n0xena]

TL;DR: only use nvme when you can afford to replace them - otherwise get cheap sata ssd - at least they're cheaper to replace