A fifth disk, then a disk replaced
Full burn-in of a second-hand disk, RAIDZ2 expanded from 4 to 5 disks without downtime, then a disk with growing bad sectors replaced, with no free bay.
Burn in before trusting
Two second-hand SAS disks arrived: one to grow the pool, the other as a spare. The first only joined the pool after a SMART long test, then a write and read-back of its whole surface (badblocks): zero bad blocks. Along the way, the burn-in revealed that the write cache was disabled on every disk: writes topped out at 23 MB/s. It is now enabled by a system rule, set so that the flushes requested by ZFS are actually passed to the disks.
RAIDZ2 from 4 to 5 disks, without downtime
ZFS can now grow a RAIDZ by adding a disk to it. The expansion copied 8.5 TB in 16 hours, with the pool online and redundant, then a full scrub found no errors: 13.6 TB raw instead of 10.9. Data already written keeps its old layout until it is rewritten: still to do, without the snapshots that would double the space used.
A disk wearing out
One of the original disks had a growing list of bad sectors: 2, then 3 during the expansion, then 4. It was replaced by the spare. With no free caddy for the server, it had to be taken offline before the swap: during the rebuild, the pool could only survive one more failure instead of two. The disk was located by its SAS address in the remote management card (which does not show serial numbers), its bay checked against the label, then 1.65 TB rebuilt in 14 h 40, without errors.
Two lessons: keep a free caddy ready, to replace a disk without reducing redundancy; and silence the “pool degraded” alert for the planned duration of an intervention, instead of letting it ring every six hours.
The rest of the week
- The secrets vault moved to a patched version, with a snapshot of the VM taken first, to be able to roll back.
- The drive wears the homelab’s colours: name, logo, background, links to the home page and the legal pages, applied by Ansible only when they change.
- Two memory incidents on the hypervisor, caused by large imports started during the day: imports and heavy disk work now happen at night.
- A page explains how AI is used in this project: AI in this project.