data loss after power failure

[shimi]

On Thu, Aug 13, 2009 at 9:59 AM, Micha Silver <micha at arava.co.il> wrote:

> We suffered some annoying file and directory corruption on a CentOS 5.3 64
> bit server two days ago after a long power failure - long enough to drain
> the UPS battery, with several short "spurts" of power until it finally
> stabilized. Files appeared as directories, directories turned into files,
> duplicated inodes, in short, a mess. And most of the corruption was under
> /usr, i.e. *not* files that were being written to when the power went off.
> The file system is ext3 but the journaling didn't help - well maybe it did,
> but not enough...
>
>
> The machine is a Dell PE 840 with their PERC 5i controller and 4 SATA disks
> in a RAID 5 array. It has its own battery backup to preserve the writeback
> cache in case of power failure (but again the files that got "kevorked" were
> not being written...). And it's relatively new ( < 2 yrs). Two identical
> machine attached to the same UPS, but w/o RAID came thru the event with no
> problems. And other, older servers with SCSI disks also showed no ill
> effects.
>
> Does anyone have any ideas how to prevent this kind of thing in the future?
> (short of adding lots of additional batteries)
>

Sample the UPS unit for "how much battery time left do you have?"  and
initiate automatic shutdown when the number falls below the 5 minutes
threshold ?

-- Shimi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20090813/a0461a50/attachment.html>