A full Gutmann

After a short delay for knee surgery, our move-out date in Munich is now less than a week away.

Since we’re planning to spend the next ~year traveling, this week I’m continuing to divest myself of things that won’t fit in a 55 liter backpack.

And that includes more than 20 hard drives I’ve used over the years: laptop, desktop, usb. With great effort, I’ve consolidated all their data onto one disk, which will be spending the next year somewhere safe and sound (it’s backed up, of course).

But what to do with all the drives? Well, some would say – smash them with a hammer and be done with them. But it would be nice if someone could make use of all these platters, would it not?  So I’m in the process of securely deleting them all so that I can give them away.

Unfortunately, securely deleting 20 drives is no easy matter. It’s not secure to just repartition and reformat – the data is still there, and may include passwords or facebook cookies or other things that could be used against me and my friends.

But, ah, you say, I’ll just zero out the drive, with a command like dd if=/dev/zero of=/dev/sdc bs=100M count=5000.

This, sadly, is quite slow, especially given that I’m erasing most of these drives with a USB/SATA adapter.

To make matters worse, writing zeroes is not good enough, according to a famous 1996 paper by Peter Gutmann. He wrote:

The problem lies in the fact that when data is written to the medium, the write head sets the polarity of most, but not all, of the magnetic domains. This is partially due to the inability of the writing device to write in exactly the same location each time, and partially due to the variations in media sensitivity and field strength over time and among devices.

In conventional terms, when a one is written to disk the media records a one, and when a zero is written the media records a zero. However the actual effect is closer to obtaining a 0.95 when a zero is overwritten with a one, and a 1.05 when a one is overwritten with a one. Normal disk circuitry is set up so that both these values are read as ones, but using specialised circuitry it is possible to work out what previous “layers” contained.

So even after you’ve zero’d a drive, or even written random data to it, the old, overwritten value can be obtained through various techniques including magnetic force microscopy and scanning probe microscopy.

Gutmann developed a 35-pass erasure algorithm, known as the Gutmann method, to thwart these techniques and eradicate every trace of the old data from a drive.

And there’s even a handy Linux command, shred, which implements the Gutmann algorithm and can be run against a file or a device node directly.

Now, Gutmann’s paper and the 35-pass erasure method are often cited, but are also at this point quite old. Hard drives have changed a lot since 1996. They’re much denser, of course. And the method by which they encode data on the disk has changed as well (PRML/EPRML vs MFM).

Plus the interesting data is now drowning in a sea of uninteresting data. My password database is a few lonely kilobytes amid gigabytes of binaries and libraries and web caches and so on. And, as several people pointed out to me just after I wrote this, recent research seems to indicate that on modern drives, one pass is enough.

So it’s overkill to pull a “full Gutmann” on these drives before donating them to the local orphanage. And I don’t have the time for that anyway.

But nevertheless, I’d feel better doing at least one pass, right? Unfortunately /dev/urandom is pretty slow for this – far slower than /dev/zero or shred, which are already un-fast. And when you’re erasing 20 drives over USB and trying to stay ahead of the moving truck, speed matters.

Well, it turns out that the ATA command set has included a built-in “secure erase” command (ATA-SE) since 2001. This command performs the entire erasure on the drive itself. Since the computer doesn’t have to shuffle bits over the (in my case) USB bus to the disk, it’s quite a lot faster (though still by no means fast – I’m currently waiting 97 minutes for a 250GB USB drive to secure-erase). Also, it erases blocks that the hard disk had marked as “bad blocks” – so those aren’t recoverable either.

Check out these instructions for using the secure erase command from Linux with hdparm.

But unfortunately there is a sad ending to this story. The ATA spec also includes a command called “freeze lock.” This command tells the drive to disable the secure-erase command. And most BIOSes issue the command to all connected ATA drives on boot.

I think my friend Phil summed this up pretty well:

phik: that's the kind of thing that makes you feel really professionally rewarded
phik: you tirelessly fight your boss to work on something, make a prototype
phik: push it through some god-awful standards body
phik: get everyone to adopt it
phik: and the bios vendors block it

(Luckily, on my thinkpad, ATA-SE is still an option. And it works on about half these USB drives. Hooray!)