First major MD failure

I just had a major failure with a raid10 md array. I thought others
might be interested, as I've never had issues with it before.

I have an external sata enclosure with two raid10 arrays (4+spare
drives and 3 drives) in it using a LSISAS1068 controller. The server
stopped responding the other day (it could only be pinged) and the
console was filled with the same rapidly scrolling scsi error (sadly I
didn't write it down...). When I hit reboot, mdadm failed to start an
array in the enclosure. None of the logs had any useful data in them,
so I don't know what actually when wrong - just the nice big XFS null
holes :-)

So anyways a cat /proc/mdadm showed only 2 of 5 drives in md0, and
that wasn't enough to rebuild the array. After poking through dev and
dmesg I determined those three missing drives weren't dead. mdadm
thought the three drives were dirty, since they were old:

# mdadm -E /dev/sd[a,e-h]1 |grep Update\ Time
    Update Time : Wed Apr 16 09:28:34 2008
    Update Time : Wed Apr 16 09:26:53 2008
    Update Time : Wed Apr 16 09:26:53 2008
    Update Time : Wed Apr 16 09:26:54 2008
    Update Time : Wed Apr 16 09:28:34 2008

So after the "crash", two drives were updated (and didn't have enough
information on them to rebuild the array), while the other three
didn't get that final sync. At this point I thought I was going to be
rebuilding a terabyte of data from scratch.

After much pondering and man page reading, I decided to see what
happen if I tried starting the array using the three older drives:

# mdadm --assemble /dev/md0 /dev/sd[efg]1 --run
mdadm: /dev/md0 has been started with 3 drives (out of 4).

And low and behold, the array started to rebuilt from the two drives
marked 9:26:53 and syncing to the 9:26:54. Not only that, I seem to
have a valid file system (xfs) on the array. I'll never know what
data I may have lost between 9:26:53 and 9:28:34, but I'm assuming not
much. The services generating data during that time period had
probably already crashed.
Received on Thu Apr 17 17:16:22 2008