Kernel memory management problem

[Valery Reznic]

--- On Mon, 3/2/09, Alexander Indenbaum <alexander.indenbaum at gmail.com> wrote:

> 
> We noticed that if we copy large media file ( 1.4G ) to
> flash storage
> filesystem, using wget/sftp/whatever kernel grabs almost
> all the
> available physical memory and buffers are  not released
> even after
> transfer is completed and sync; sync; sync is issued :). It
> constitutes a problem - applications have no enough memory
> to run -
> crash and burn in hell. See bellow memory consumption
> numbers (top
> output):
Are you really experience some problem or just worried
about what free is shows ?

To me it's looks perfectly normal:
almost all of your memory is cached.

I.e kernel know, that this memory was read/written to/from file(s)
and if any of this content is needed kernel can provide it without access to the filesystem (disk, flash, whatever).
But on the other size when kernel ready to give up that memory, when real need arise.

Valery. 


> 
> Before test start: Mem: 97296K used, 807096K free, 0K shrd,
> 1732K
> buff, 54712K cached
> During/After test: Mem: 897908K used, 6484K free, 0K shrd,
> 2872K buff,
> 835088K cached
> 
> It worth mentioning that probably we see this issue since
> network IO
> rate is higher than storage IO rate. I’m not sure which
> kernel
> subsystem is memory grabber: VFS buffer cache, storage
> driver maybe
> something else? Bottom line is we don’t want to let the
> kernel to be
> so memory hungry and it’s desirable to set some
> reasonable limit on
> memory usage and thus work around this problem.
> 
> What are your recommendations?
> 
> ~baum
> 
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il