Kernel memory management problem

[guy keren]

if you don't care about the speed of copying the data and of slowing 
down the disk media consderably during this copying, try to change the 
copy program, so it will use the O_DIRECT flag when opening the file 
(and then you'll need to make sure the buffers you pass to write() are 
aligned to disk sector size, as well as being in disk sector size 
multiples). this way, you'll bypass the buffer cache altogether.

maybe someone else will come up with a better response.

--guy

Alexander Indenbaum wrote:
> Guys/Girls/Dudes,
> 
> I'm working with embedded Linux box powered by 2.4 kernel ( I know :)
> we are in process of switching to 2.6 ) with 1G physical memory, no
> swap defined and some slow flash storage mounted.
> 
> We noticed that if we copy large media file ( 1.4G ) to flash storage
> filesystem, using wget/sftp/whatever kernel grabs almost all the
> available physical memory and buffers are  not released even after
> transfer is completed and sync; sync; sync is issued :). It
> constitutes a problem - applications have no enough memory to run -
> crash and burn in hell. See bellow memory consumption numbers (top
> output):
> 
> Before test start: Mem: 97296K used, 807096K free, 0K shrd, 1732K
> buff, 54712K cached
> During/After test: Mem: 897908K used, 6484K free, 0K shrd, 2872K buff,
> 835088K cached
> 
> It worth mentioning that probably we see this issue since network IO
> rate is higher than storage IO rate. I’m not sure which kernel
> subsystem is memory grabber: VFS buffer cache, storage driver maybe
> something else? Bottom line is we don’t want to let the kernel to be
> so memory hungry and it’s desirable to set some reasonable limit on
> memory usage and thus work around this problem.
> 
> What are your recommendations?
> 
> ~baum
> 
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il