Read-Only Is a Superpower: Recovering a Dying NVMe Over SSH

A small tax firm called me on a Monday. Their server had “reset itself” over the weekend. Everything looked wiped.

It hadn’t reset. The truth was worse, and more interesting.

The system NVMe, a cheap DRAM-less consumer drive that had no business running a production database around the clock, was dying. On the last reboot the BIOS couldn’t see it, shrugged, and booted an ancient Windows off a second disk nobody remembered was still in the box. To the staff it looked like the machine had been factory-reset. To me it looked like a drive with unreadable sectors and a controller that crashed the moment you leaned on it.

The data, a DATEV install with a SQL Server behind it, was still on the dying disk. Somewhere.

Here is how I got it back. Almost entirely over SSH, with a live USB, and an AI copilot doing the boring parts so I didn’t fat-finger the one command you don’t get to take back.

Rule zero: never touch the patient

A dying disk gets exactly one chance to cooperate. Every write, every filesystem check, every accidental boot is a coin flip you don’t need to make.

So the box never boots its own Windows again. It boots grml, a Debian-based live Linux, off a USB stick, in forensic mode. The dying NVMe gets mounted read-only and stays that way. Belt and suspenders:

nvme_core.max_retries=0
blockdev --getro /dev/nvme0n1   # must print 1, forever

If that number is ever anything but 1, you stop.

A live USB is quietly one of the best tools in the trade. It turns any dead machine into a trustworthy, disposable, read-only workstation with a full Linux userland. No install, no footprint, no arguing with the OS that’s already broken.

The jumpbox collapses distance

Here’s the part people underestimate: I did most of this from my desk, in another city.

There is one small always-on Debian box sitting in that office’s LAN. A jumpbox. Once you can ssh to it, you’re inside the network, and from there you can reach everything else. The live-USB machine came up with DHCP, started its SSH daemon, and suddenly the patient was a hop away.

The client’s own internet flaked. I was tethering off a phone. None of it mattered, because the long-running job lived in a screen session on the box, and I reached the box through the jumpbox, not through my flaky link. Connections dropped. The rescue didn’t notice.

Distance is a solved problem. You just have to build the one bridgehead before you need it.

ddrescue is patience with a map

ddrescue clones a failing disk the smart way: it grabs the easy, healthy regions first, keeps a map of what it got, and circles back for the hard parts later. You can stop, reboot, cool the drive down, and resume exactly where you left off, because the map file is the memory.

The whole job was 98% done from earlier passes. What remained was the scary 2%: a 16 GB “crash zone” that had twice hung the controller hard enough to need a power cycle. The trick is to read it backwards:

ddrescue -f -d -n -T 2m -R -i <gap-start> -s <gap-size> \
  /dev/nvme0n1 /dev/sda /mnt/hdd/nvme.map

Reading from the top of the gap downward means you recover the bulk before you ever approach the exact sector that kills the controller. My AI copilot watched the whole run: rate, position, error count, controller state, tailing the log every minute, ready to tell me the instant I needed to physically pull power.

I never needed to. A rested, cooled drive read the sector that had crashed it twice. The gap closed to zero. Clone: 100%.

That’s the emotional high of this work. A number that was stuck for days finally hits 100.

Naming the dead

66 kilobytes didn’t come back. Sixteen tiny dead sectors, truly gone.

The question that matters isn’t “how many sectors died,” it’s “which files did they land in.” So you translate: bad sector to partition offset, offset to NTFS cluster, cluster to inode, inode to path. Tedious, error-prone arithmetic, exactly the kind of thing you want a machine to do while you double-check it:

ifind -o <part-start> /dev/sda -d <cluster>   # cluster -> inode
ffind -o <part-start> /dev/sda <inode>        # inode  -> path

The verdict: two backup PDFs, a couple of installers, some backup-repo chunks, a recycle-bin leftover. Nothing in the Windows system files, nothing in the registry, nothing in the live databases. The clone was, for all practical purposes, whole.

The one file that wasn’t in the backup

There was a verified backup on an external SSD. “Verified” as in: someone had checked it copied. This is where it gets uncomfortable.

I diffed the backup against the clone, file by file. The backup was complete except for exactly one file. And that one file was, of course, the one file with a dead sector inside it on the clone.

The single file missing from the backup was the single file physically destroyed on the disk.

Cosmic timing. It turned out to be a legacy 2007 database that isn’t even attached anymore, so it didn’t matter operationally, and a copy lives in an older backup because it hasn’t changed since 2007. But sit with the shape of it: a backup you have not restore-tested is not a backup. It’s a hopeful copy. The gap is always exactly where you’re not looking.

Booting the ghost, verifying without the keys

New disk, clone written to it, defective drive pulled and bagged for warranty. First boot, offline, no network, nothing that could phone home or auto-update.

It came up. Filesystem clean, no repair needed. SQL Server started on its own and ran crash recovery.

Now, how do you confirm 71 databases are healthy when you don’t have the SQL admin password? You don’t need it. SQL Server writes everything to a plain text error log, and as a local admin you can just read it. Every database reported a clean consistency check. No corruption errors. The one damaged file wasn’t even attached.

To do that reading remotely I turned on WinRM, tunneled to it through the same jumpbox, and let the copilot parse the log. No guessing, no “looks fine,” just the engine’s own words: 71 databases, clean.

The firm was working again by mid-morning.

What actually did the work

No single hero here. A stack of boring, composable tools, each doing one thing well:

• A live USB turned a broken box into a safe, read-only Linux workstation.
• A jumpbox turned “another city” into “one SSH hop away.”
• ddrescue turned a dying disk into a resumable, mappable, patient copy.
• The Sleuth Kit turned dead sectors into filenames.
• An AI copilot watched the long jobs, did the arithmetic, ran the lookups, read the logs, and held the entire state in its head across every dropped connection, so I could stay focused on the two or three decisions that actually needed a human.

That last one is the shift I keep noticing. The tools haven’t changed much in twenty years, and they’re still the right tools. What changed is that I now have a tireless operator sitting between me and them, one that never loses the thread, never mistypes the destination of a dd, and never gets bored on hour three of a rescue.

Read-only discipline, a bridge into the network, and a copilot that keeps state. That’s the whole job.

And restore-test your backups. The one file you skip is the one the universe wants.