Cannot remove log device from pool

[lowjoel]

Related: #8748

Recently I've had an SLOG SSD brown-out on me and I've needed to force detach the device. I've since replaced the single SLOG with a mirrored SLOG, but I can't remove the old device. I've temporarily worked around the missing SLOG by zpool offline -f <pool> <dev> but removing it is probably the right thing to do.

All my attempts have been rebuffed by zpool no matter where from: initramfs, in a recovery boot, or under normal situations. It returns me cannot remove <guid>: Mount encrypted datasets to replay logs. despite having zfs mount -a.

The last request for information was #8748 (comment) where logs were requested. I've done that on my machine, it's quite long but the lines leading up to the -EBUSY look like:

1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:545:zap_entry_read_name(): error 75
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 zap_micro.c:1137:zap_lookup_impl(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 zap_leaf.c:461:zap_leaf_lookup(): error 2
1751783415   ffff98bd92fd5200 dsl_prop.c:178:dsl_prop_get_dd(): error 2
1751783415   ffff98bd92fd5200 zil.c:4068:zil_suspend(): error 16
1751783415   ffff98bd92fd5200 zil.c:4367:zil_reset(): error 16
1751783415   ffff98bd921dd200 zap.c:857:fzap_checksize(): error 22
1751783415   ffff98bdc1a88000 dsl_dir.c:1344:dsl_dir_tempreserve_impl(): error 28
1751783415   ffff98bdc1a88000 dmu.c:478:dmu_spill_hold_existing(): error 2
1751783415   ffff98bdc1a88000 sa.c:370:sa_attr_op(): error 2
1751783415   ffff98bdc1a88000 zfs_dir.c:1219:zfs_get_xattrdir(): error 2
1751783415   ffff98bdc1a88000 dsl_dir.c:1344:dsl_dir_tempreserve_impl(): error 28
1751783415   ffff98bdc1a88000 dmu.c:478:dmu_spill_hold_existing(): error 2
1751783415   ffff98bdc1a88000 sa.c:370:sa_attr_op(): error 2
1751783415   ffff98bdc1a88000 zfs_dir.c:1219:zfs_get_xattrdir(): error 2
1751783415   ffff98bdc1a88000 dsl_dir.c:1344:dsl_dir_tempreserve_impl(): error 28

This is running zfs 2.3.3 (dkms) on Ubuntu 24.04. Any help to debug this would be appreciated (and hopefully we can find a fix; otherwise I'd have to replicate my entire pool which may not be a bad idea...)

[lowjoel]

Today I tried running zpool remove rpool <device uuid> again, observations:

• The command seemed to stall for several minutes
• zpool_influxdb, l2arc_feed and other zfs commands running in parallel seemed to have been uninterruptible (possibly deadlocked?)
• after ~10 minutes I panicked and rebooted the system. It eventually rebooted and the errored SLOG device was removed after reboot.

I can't explain it, and I coulda swore this exact same sequence happened the earlier few times without me succeeding after a reboot. I'll close this because it'll be unreproducible for me; if you're still affected please open another discussion.