Get a Quote     (408) 943-4100               TrueNAS Discord      VendOp_Icon_15x15px   Commercial Support Toggle between Light and Dark mode
TrueNAS.com Software Systems Company Community Security iX Portal Download

  1. Documentation Hub
  2. /
  3. References
  4. /
  5. L2ARC
Edit this page

L2ARC

  3 minute read.

Last Modified 2022-08-24 12:49 -0400

ZFS has several features to help improve performance for frequent access data read operations. One is Adaptive Replacement Cache (ARC), which uses the server memory (RAM). The other is second level adaptive replacement cache (L2ARC), which uses cache drives added to ZFS storage pools. These cache drives are multi-level cell (MLC) SSD drives and, while slower than system memory, are still much faster than standard hard drives. ZFS (including TrueNAS) uses all of the RAM installed in a system to make the ARC as large as possible, but this can be very expensive. Cache drives provide a cheaper alternative to RAM for frequently accessed data.

How Does L2ARC Work?

When a system gets read requests, ZFS uses ARC (RAM) to serve those requests. When the ARC is full and there are L2ARC drives allocated to a ZFS pool, ZFS uses the L2ARC to serve the read requests that overflowed from the ARC. This reduces the use of slower hard drives and therefore increases system performance.

Implementation in TrueNAS

TrueNAS integrates L2ARC management in the web interface Storage section. Specifically, adding a Cache vdev to a new or existing pool and allocating drives to that pool enables L2ARC for that specific storage pool.

Cached drives are always striped, not mirrored. To increase an existing L2ARC size, stripe another cache device with it. You cannot share dedicated L2ARC devices between ZFS pools.

A cache device failure does not affect the integrity of the pool, but it may impact read performance depending on the workload and the dataset size to cache size ratio.

Persistant L2ARC in CORE and SCALE

By default, the L2ARC cache empties when the system reboots. When Persistent L2ARC is enabled, a sysctl repopulates the cache device mapping during the reboot process. Persistent L2ARC preserves L2ARC performance even after a system reboot.

However, persistent L2ARC for large data pools can drastically slow the reboot process, degrading middleware and web interface performance. Because of this, we’ve disabled Persistent L2ARC by default in TrueNAS CORE, but you can manually activate it.

Activating Persistent L2ARC

Go to System > Tunables and click ADD. For the Variable, enter vfs.zfs.l2arc.rebuild_enabled. Set the Value to 1 and the Type to sysctl. We recommend noting in the Description that this is the Persistent L2ARC activation. Make sure Enabled is set and click SUBMIT.

PersistentL2ARCTunable

TrueNAS CORE doesn’t write settings changed through the CLI to the configuration database. TrueNAS will reset them on reboot.

In a command line, enter sysctl vfs.zfs.l2arc.rebuild_enabled=1. When successful, the output reads: vfs.zfs.l2arc.rebuild_enabled: 0 -> 1

TrueNAS SCALE enables Persistent L2ARC by default. We do not recommend users disable it.

Device Recommendations

Like all complicated features, deciding whether L2ARC is effective or not requires a strong understanding of your storage environment, performance goals, and the software you’re using.

However, we have a few recommendations for L2ARC devices:

  • Using multiple L2ARC devices helps reduce latency and improve performance.

  • Random Read Heavy workloads can benefit from large capacity L2ARC SSDs. L2ARC SSDs are faster than the existing data storage drives.

  • Sequential or streaming workloads need very fast, low-latency L2ARC devices. We recommend Enterprise-grade NVMe devices. L2ARC device capacity depends on how much faster it is than the data storage devices. An L2ARC device that is much faster than the data storage devices makes better use of its larger capacity.

Resources