Strange problem

[Geoff Shang]

Hi,

I was wondering if anyone has any thoughts on the following which happened 
to a virtual server that's being used for a project I'm involved with.

Geoff.




---------- Forwarded message ----------

This isn't the usual sort of crash that we were experiencing with VPSLink; it 
doesn't seem to be usage related. However, the problem is *extremely* strange. 
First, attempting to read /proc/987/cmdline was hanging. (Pid 987 was udevd.) 
As I understand it, this should never happen, but as a result, ps, pgrep, etc. 
were all just hanging when I ran them. This could have been causing all sorts 
of hell. (I figured this out using strace on ps.) Once I killed this off, ps 
eventually started behaving again, but apache was still hung. I managed to 
restart apache, but it was still misbehaving. Wanting to get the box up as soon 
as possible, I then just rebooted the box.

Any ideas? This would seem to suggest some sort of kernel level issue; I can't 
see why /proc/<pid>/cmdline should ever just hang like that. I can find other 
info on ps hanging due to /proc/<pid>/status, but can't seem to find anything 
about cmdline causing the hang.

Anyway, box is back up now.

I think I'm going to write a script running in a cron job on a different box to 
make sure we get notified if this goes down again. It hasn't gone down in 6 
weeks, so I suspect this is a once off, but it's still bad.