I Have a "home lab" system (Ubuntu 22.04 on i3-9300 with 32G ecc-ram, 10G of witch is dedicated to hugepages) currently holding 2 zfs raidz1 pools, one with 3x2TB WD-RED and nvme as log and cache device, and the second one with 3x1TB HDD (planing to extend this one to 3x12TB) currently used as backup only
bartoszek@home-lab:~$ zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
storage-pool 2.72T 585G 2.15T - - 2% 21% 1.00x ONLINE -
raidz1-0 2.72T 585G 2.15T - - 2% 21.0% - ONLINE
sdb - - - - - - - - ONLINE
ata-WDC_WD10EFRX-68FYTN0_WD-WCC4J7LNUNJS - - - - - - - - ONLINE
ata-WDC_WD10EZEX-00RKKA0_WD-WMC1S5929034 - - - - - - - - ONLINE
system-pool 5.45T 3.46T 1.99T - - 3% 63% 1.00x ONLINE -
raidz1-0 5.45T 3.46T 1.99T - - 3% 63.5% - ONLINE
ata-WDC_WD20EFRX-68EUZN0_WD-WMC4M1371094 - - - - - - - - ONLINE
ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M6HSDJXY - - - - - - - - ONLINE
ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M1628970 - - - - - - - - ONLINE
logs - - - - - - - - -
nvme0n1p2 15.5G 1.42M 15.5G - - 0% 0.00% - ONLINE
cache - - - - - - - - -
nvme0n1p3 64.0G 54.6G 9.44G - - 0% 85.2% - ONLINE
The whole system is used mainly for Nextcloud, Plex, zoneminder and some Virtualization (GPU passthrough to a Windows10 vm, some other stuff) so nothing really special (please notice no torrent)
Because some of my use cases require load of ram to work (VM/Zoneminder) I have limited resources for ZFS caches (and as for now no plan to extend RAM) so I want to squeeze as much performance as possible from the HW I currently own. The workload I have can be split into basically two main categories, write a lot of data/read almost none of it or read once (zoneminder/plex/backups) and write once/read multiple times (games on VM/nextcloud) I decided to disable cache for first group and enable for rest. So to recap
- Nextcloud -> primarycache=all secondarycache=all
- VM -> primarycache=all secondarycache=all
- rest of stuff-> primarycache=none secondarycache=metadata
But then I realize, that the second category can be divided further to 'I need it as fast as possible' (Nextcloud) and 'just don't be annoyingly slow' (Ocasional gaming on my vm). And just to be clear, nextcloud is NOT used more often I just want to prioritize ARC to hold as much Nextcloud data in ram as possible and in the same time cache game data on NVME, because more annoying for me is slow nextcluod, then a little bit longer loading times of some games. And started to make some tests by creating a temporary dataset
sudo zfs create -o recordsize=1M -o primarycache=none -o secondarycache=all -o mountpoint=/srv/tmp_1M_noprimarycache system-pool/ENC/tmp_1M_noprimarycache
and then to put some read only stress on it I tried
sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=1024k --iodepth=2 --size=4G --readwrite=randread --loops=10
And noticed that my cache device is NEWER used (zpool iostat shows no IO on NVME drive) I assume that this is because L2ARC is feed by "drops" from main ARC but why is this configuration even valid? And how to configure ZFS to use L2ARC without main ARC?
