md device failure (help!)

[Shachar Tal]

Hi all,
my md array just crapped out on me! I'm partly responsible, since one of the
device in the RAID5 array died some time ago and I neglected to replace it,
but I don't think it's the whole problem now.

When I assemble the array I get the following:
root at server:~# mdadm --assemble --verbose /dev/md0 /dev/sdb1 /dev/sdc1
/dev/sdd1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 3.
mdadm: no uptodate device for slot 0 of /dev/md0
mdadm: added /dev/sdb1 to /dev/md0 as 1
mdadm: added /dev/sdd1 to /dev/md0 as 3
mdadm: added /dev/sdc1 to /dev/md0 as 2
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array

(slot 0 is the long-dead drive)
the output of "mdadm --examine" for 2 of the drive (sdc & sdd) is similar
and looks like this:
...
State: Clean
Active Devices: 2
Working Devices: 2
Failed Devices: 1
Events: 1923796
...

while the output for sdb looks different:
...
State: active
Active Devices: 3
Working Devices: 3
Failed Devices: 0
Events: 1923787
...

Note the difference in the Events counter and the state. My guess is that
the drive is out of sync with the rest.
I tried "mdadm --assemble --force --update=summaries" to bring the stray
Events counter up to date per a recommendation I saw in a forum, but this
command segfaults.
I tried strace-ing it and it faults right after reading 4K of data from
/dev/sdb1.

To summarize: I'm not sure what to do next. I've read in forums that I
should try to re-create the array but I fear it will completely destroy the
data (not sure what creating an array from previously-array-ed disks does).

Any help will be appreciated. really!

Thanks,
-- 
-- Shachar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20090703/443febc7/attachment.html>