ZFS performance on NVME drives

[blind-oracle]

Hi all.

Probably a general question, though I wanted to ask what can limit the ZFS performance on NVME drives.

I have a test rig that consists of 6 rather slow-ish NVME disks that can do maybe 2-3GB/s linear reads each.
On each device I have a LUKS encryption and then ZFS pool on top of LUKS device-mapper devices - this is all due to the fact that built-in ZFS encryption is still 3-4x time slower than LUKS sadly.

The pool is raidz1.
Overall this works very well and fast, but when doing scrub I see that its performance is limited to around 4GB/s:

# zpool status
  pool: zfs
 state: ONLINE
  scan: scrub in progress since Mon Nov  3 21:06:33 2025
        8.85T / 16.8T scanned at 24.9G/s, 1.34T / 16.8T issued at 3.78G/s
        0B repaired, 8.01% done, 01:09:42 to go

(this grows to 4GB/s+ eventually)

The CPU load in process is quite low, around 90% is idle and no single core is saturated (CPU is Ryzen 9 9900x with DDR5-ECC @ 4200Mhz)

iostat shows that ZFS reads evenly around 700MB/s from each device and that's it.

Not that I need higher performance (network is 10Gbit anyway), but more like out of curiosity - what can be the bottleneck here? Memory bandwidth/latency? Doesn't look like.

Thanks in advance. Ah, yes, OpenZFS is latest 2.3.4 on Ubuntu 24.

[amotin]

I don't know what can be a limiting factor in your case, just keep in mind that scrub speeds reported by zpool status may be inaccurate if ZFS has to switdh between scan and issue several times, though if it matches iostat output, it might be not the problem. Generally though, after some optimizations few years ago, scrub is now one of the fastest operations, which on a big and fast hardware shown me something up to ~60GB/s IIRC, so 4GB/s is definitely not a hard limit. Since you've mentioned LUKS, it makes me wonder if some limits might be there, since for native ZFS encryption scrub does not need to decrypt the data, while if encryption implemented below ZFS -- it has have to.

[blind-oracle]

Thanks a lot for the input!

Yes, zpool status roughly matches iostat's sum of all disk's bandwidth, so probably I can trust it.
As for LUKS, I've checked that it's not the issue - I can read from all encrypted drives at the same speed as from the raw ones, with negligible CPU load (10-15%):

# dd if=/dev/mapper/NVME01 of=/dev/null bs=32M iflag=direct
4459+0 records in
4458+0 records out
149585657856 bytes (150 GB, 139 GiB) copied, 50.8038 s, 2.9 GB/s

(and 5 more in parallel the same).

Without direct mode, though, the speed goes down to around 1GB/s, but I'm pretty sure that ZFS bypasses the page cache when accessing the drives, right?

# cat /sys/module/zfs/parameters/zfs_dio_enabled
1

[amotin]

All synthetic tests are only valid until they are not. ZFS never uses 32MB I/Os.

[blind-oracle]

Makes sense, thanks.

Yes, with e.g. 128k IOs (I have a default recordsize) the sequential read with these NVMEs drops to around 1.4GB/s which is probably closer to what I can expect from a scrub when multiplied by a number of drives, but still a bit low-ish I guess: 1.4 * 6 (or 5 in case of raidz?) should be closer to 7-9GB/s than 4GB/s.

[1JorgeB]

For another data point, this is with 7 NVMe devices, they show around 2GB/s reads per device during a scrub, hardware is nothing special:

AMD EPYC 7232P 8-Core @ 3100 MHz
128 GiB DDR4 ECC @ 3200MT/s