data loss after power failure

[Micha Silver]

We suffered some annoying file and directory corruption on a CentOS 5.3 
64 bit server two days ago after a long power failure - long enough to 
drain the UPS battery, with several short "spurts" of power until it 
finally stabilized. Files appeared as directories, directories turned 
into files, duplicated inodes, in short, a mess. And most of the 
corruption was under /usr, i.e. *not* files that were being written to 
when the power went off. The file system is ext3 but the journaling 
didn't help - well maybe it did, but not enough...


The machine is a Dell PE 840 with their PERC 5i controller and 4 SATA 
disks in a RAID 5 array. It has its own battery backup to preserve the 
writeback cache in case of power failure (but again the files that got 
"kevorked" were not being written...). And it's relatively new ( < 2 
yrs). Two identical machine attached to the same UPS, but w/o RAID came 
thru the event with no problems. And other, older servers with SCSI 
disks also showed no ill effects.

Does anyone have any ideas how to prevent this kind of thing in the 
future? (short of adding lots of additional batteries)


Thanks,

Micha