stack/hw/RUNBOOK.md

Storage Node Build Runbook

Hardware: ASUS ROG Crosshair VIII Hero / AMD 3950X / 192GB RAM OS: Alpine Linux 3.21 (custom ISO) Security: LUKS2 encryption + YubiKey FIDO2 SSH + YubiKey challenge-response unlock

Drive Map

Device	Hardware	Mount	Encryption	Purpose
Samsung Fit 128GB USB	Rear USB 3.1 port	/ (read-only)	None (no secrets)	OS root
6x WD Blue SA510 2TB	M.2→SATA adapters on SATA6G_1-6	/data/rustfs (LVM, 12TB)	LUKS2 on LV	RustFS object storage
1x Modern NVMe 2TB	M.2_1 onboard slot (PCIe 3.0 x4)	/var/lib/docker + /var/cache	LUKS2 per partition	Docker, writes, cache

Security Model

Boot → read-only root (no secrets, no data) → SSH only with YubiKey FIDO2
  → yubikey-unlock (challenge-response decrypts LUKS volumes)
  → Docker starts → RustFS serves data

• At rest: All data encrypted with LUKS2 (AES-XTS-512). Server can be physically stolen and data is safe.
• SSH access: Requires a hardware YubiKey with FIDO2 ed25519-sk key. No passwords accepted.
• Volume unlock: Requires physical YubiKey challenge-response (HMAC-SHA1 slot 2).
• Backup access: Emergency passphrase enrolled in LUKS slot 1 (store offline, safe deposit box).
• Two YubiKeys enrolled: Primary and backup, both work for SSH and LUKS.

YubiKey Preparation (before build)

On your workstation, for EACH YubiKey:

# 1. Program HMAC-SHA1 challenge-response on slot 2
ykman otp chalresp --touch --generate 2

# 2. Generate FIDO2 SSH key (resident on the key)
ssh-keygen -t ed25519-sk -O resident -C "yubikey-1-storagenode" -f ~/.ssh/id_yubikey1_storagenode

# Repeat with second key:
ssh-keygen -t ed25519-sk -O resident -C "yubikey-2-storagenode" -f ~/.ssh/id_yubikey2_storagenode

Save both .pub files — you'll paste them into /root/.ssh/authorized_keys during setup.

BIOS Settings

Enter BIOS: hold Delete during POST.

Required

1. Advanced → Onboard Devices Configuration

   • M.2_2 PCIe Bandwidth Configuration → Disabled(X8 mode)
   • HD Audio Controller → Disabled
   • RGB LED lighting (working state) → Off
   • RGB LED lighting (sleep/off) → Off

2. Advanced → CPU Configuration

   • SVM Mode → Enabled (AMD-V, for Docker/future VMs)

3. Advanced → AMD fTPM configuration

   • Firmware TPM → Enabled

4. Advanced → SATA Configuration

   • SATA Mode → AHCI
   • Verify all 6 SATA ports show Enabled

5. Advanced → APM Configuration

   • Restore On AC Power Loss → Power On
   • Power On By PCI-E/PCI → Enabled (Wake-on-LAN)

6. Extreme Tweaker

   • TPU → TPU II (water cooling overclock profile)

7. Boot

   • Boot priority: USB drive first
   • CSM (Compatibility Support Module) → Disabled (pure UEFI)
   • Fast Boot → Disabled (until stable)

Optional Performance

8. Extreme Tweaker → PBO (if available in BIOS update)
   • Precision Boost Overdrive → Enabled

Build Steps

Phase 1: Build the ISO (on your current machine)

cd ~/stack/alpine-iso
chmod +x build-iso.sh
./build-iso.sh

If building from a non-Alpine system:

docker run --rm -v $(pwd):/work -w /work alpine:3.21 sh -c "
  apk add alpine-sdk build-base alpine-conf syslinux xorriso \
    mtools dosfstools grub grub-efi squashfs-tools git sudo && \
  adduser -D build && \
  addgroup build abuild && \
  echo 'build ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \
  su build -c 'abuild-keygen -an' && \
  chmod +x build-iso.sh && \
  ./build-iso.sh
"

Phase 2: Write ISO to a temporary USB

dd if=out/alpine-storagenode-*.iso of=/dev/sdX bs=4M status=progress
sync

Phase 3: Boot and install to Samsung Fit

1. Plug BOTH the temp ISO USB and the Samsung Fit into rear USB ports
2. Boot from the ISO USB
3. Log in as root (no password)
4. Run: setup-alpine (use defaults, skip disk install)
5. Run: chmod +x install-to-usb.sh && ./install-to-usb.sh
6. Remove the ISO USB, reboot

Phase 4: Configure storage + encryption

1. Boot from Samsung Fit
2. Log in as root (still has password access for initial setup)
3. Have YubiKey #1 inserted
4. Run: chmod +x /root/storage-setup.sh && /root/storage-setup.sh
   • This creates the LVM volume group, encrypts all data volumes, enrolls both YubiKeys
   • You'll be prompted for a backup passphrase — store this OFFLINE
5. Add your SSH public keys:

   nano /root/.ssh/authorized_keys
   # Paste both yubikey .pub lines
   

6. Edit the RustFS password:

   nano /opt/rustfs/docker-compose.yml
   

7. Lock it down:

   ro
   

Phase 5: Test the full boot cycle

1. Reboot the server
2. From your workstation: ssh -i ~/.ssh/id_yubikey1_storagenode root@<ip>
   • Touch YubiKey when it blinks (FIDO2 auth)
3. On the server: yubikey-unlock
   • Touch YubiKey again (challenge-response to decrypt LUKS)
4. Start RustFS: cd /opt/rustfs && docker compose up -d

Phase 6: Verify

# LUKS status
dmsetup ls
cryptsetup status docker-crypt
cryptsetup status rustfs-crypt

# LVM status
pvs
vgs
lvs

# Check mounts
mount | grep -E '(docker-crypt|cache-crypt|rustfs-crypt|tmpfs)'

# Docker
docker ps
docker logs rustfs

# Root is read-only
touch /testfile    # should fail: "Read-only file system"

# RustFS health
curl http://localhost:9000/minio/health/live

# SMART status
for d in /dev/sd?; do smartctl -H "$d"; done
smartctl -H /dev/nvme0n1

Daily Operation

Power on → server boots to locked state (SSH only)
         → SSH in with YubiKey
         → yubikey-unlock (decrypts data, starts Docker)
         → RustFS serving

Power off / reboot → data re-encrypted automatically

Maintenance

# System updates
sys-update          # handles rw/ro automatically

# LVM status
pvs
vgs
lvs

# Add a new SATA drive to expand storage
pvcreate /dev/sdX
vgextend rustfs-vg /dev/sdX
lvextend -l +100%FREE /dev/rustfs-vg/rustfs-lv
# Unlock rustfs-crypt first, then grow the filesystem live:
xfs_growfs /data/rustfs

# Replace a failed SATA drive
# 1. Move data off the dying drive:
pvmove /dev/sdX
# 2. Remove from VG:
vgreduce rustfs-vg /dev/sdX
pvremove /dev/sdX
# 3. Swap physical drive, then add the new one:
pvcreate /dev/sdY
vgextend rustfs-vg /dev/sdY

# Docker management (volumes must be unlocked)
cd /opt/rustfs
docker compose logs -f
docker compose restart
docker compose pull && docker compose up -d

# Manually lock volumes (before maintenance/travel)
yubikey-lock

Emergency Recovery

If both YubiKeys are lost/destroyed:

# Boot server, SSH will fail (no valid keys)
# Connect keyboard + monitor directly
# Log in as root (if password still set) or boot from ISO
# Use backup passphrase to unlock:
vgchange -ay rustfs-vg                        # activate LVM first
cryptsetup open /dev/rustfs-vg/rustfs-lv rustfs-crypt  # enter backup passphrase
cryptsetup open /dev/nvme0n1p1 docker-crypt  # enter backup passphrase
cryptsetup open /dev/nvme0n1p2 cache-crypt   # enter backup passphrase
mount /data/rustfs
mount /var/lib/docker
mount /var/cache

Network (post-install)

rw
cat > /etc/network/interfaces << 'EOF'
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 192.168.1.X/24
    gateway 192.168.1.1
EOF
echo "nameserver 1.1.1.1" > /etc/resolv.conf
ro
reboot