Kernel memory management problem

[Alexander Indenbaum]

On Mon, Mar 2, 2009 at 5:00 PM, Valery Reznic <valery_reznic at yahoo.com> wrote:
>
> --- On Mon, 3/2/09, Alexander Indenbaum <alexander.indenbaum at gmail.com> wrote:
>
>> From: Alexander Indenbaum <alexander.indenbaum at gmail.com>
>> Subject: Re: Kernel memory management problem
>> To: valery_reznic at yahoo.com
>> Cc: "linux-il." <linux-il at cs.huji.ac.il>
>> Date: Monday, March 2, 2009, 4:45 PM
>> On Mon, Mar 2, 2009 at 4:35 PM, Valery Reznic
>> <valery_reznic at yahoo.com> wrote:
>> >>
>> >> Unfortunately this is real problem: if you do
>> something
>> >> like
>> >> while(true) { transfer } then application crashes
>> and burns
>> >> in hell :)
>> > Are you sure that culprit is kernel ?
>> > May be "transfer" part has memory leak ?
>> >
>> > Did you try to run it under valgrind ?
>> > or run top ?
>>
>> Well, I have to fix myself. Eventually memory is reclaimed.
>> The
>> "transfer" applications do not grow, while
>> observed in top. The
>> "transfer" applications are from standard Unix
>> tool set, so while bugs
>> are possible I personally blame Linus :).
> It's your right. The question is how effective it will help to solve your problem :)
>
> Do you run a standard kernel or a custom one ?
> Is your transfer program run alone or there are other applications
> running ?
> Kernel has parameter, something like "total over commit" - when
> treshold  is reached even when there is a lot of FREE memory any single malloc or mmap will fail.
>

We're using pretty standard 2.4.36 kernel.

I looked at total over commit, a.k.a. vm.overcommit_memory sysctl parameter.
It's set to 0 by default on my system. From description I've
harvested on Google (http://www.win.tue.nl/~aeb/linux/lk/lk-9.html):
It is described in Chapter: Going in the wrong direction :) on old
kernels. There is also chapter  "Going in the right direction" which
describes newer  vm.overcommit_ratio. My kernel does not provide such
interface.

Yet I tried to play with values of vm.overcommit_memory and set it to
1, 2, 80 and did not notice any improvement - free still drops till
about 6MiB. I'm also not sure it is relevant to my case since
"transfer" application does not grow at all and stays bellow 10MiB, so
it does not malloc() big chunks. I'm getting more and more sure it is
VFS buffer cache problem. For instance quick way to reclaim the cached
memory is removing the 1.4G file from the filesystem.

Yet I've found some patch which looks promising from description point
of view: http://mailman.uclinux.org/pipermail/uclinux-dev/2007-January/041671.html
I'm not an expert on VM subsystem and not sure it will apply cleanly on 2.4.36

Any thoughts?

> Valery.
>