<div dir="ltr"><div>Thanks Orna,</div><div><br></div>My understanding is, that in stock Linux kernel, a process that allocates too much memory is unlikely to receive NULL from malloc. The more likely scenario is, the whole system would swap out pages, and the OOM killer would, hopefully, kill the offending process. During that time it is unlikely that malloc would return NULL.<div><br></div><div>Indeed, if you look at actual applications written for Linux, you'd see that many use variants of xmalloc, which simply aborts if malloc returns NULL. I think that the logic is similar to what I wrote before.</div><div><br></div><div>I thought to give a list of such real world programs, but someone wrote it much better than I did:</div><div><br></div><div><a href="http://eli.thegreenplace.net/2009/10/30/handling-out-of-memory-conditions-in-c">http://eli.thegreenplace.net/2009/10/30/handling-out-of-memory-conditions-in-c</a></div><div><br></div><div>But feel free to correct me if I'm wrong.<br><div><br></div><div>PS,</div><div>I think we have miscommunication here, since from what I understand the paper linked is about memory allocation for a VM, while I'm talking about memory allocation for a process.</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, May 16, 2015 at 10:10 PM, Orna Agmon Ben-Yehuda <span dir="ltr"><<a href="mailto:ladypine@gmail.com" target="_blank">ladypine@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Elazar,<div><br></div><div>I find that malloc failure checking is vital (within the user program) even on a regular system with gigabytes of memory. For example, when the program gets into a recursive loop which allocates memory and then digs deeper. In other cases, It is useful to check the return value of malloc when the program input size is unlimited, and it is better to inform the user about the too-large element of the input rather than to crash.<div><br></div><div>I do not understand, however, how memory overcommitment leads you to not require malloc to fail. Maybe when you say overcommitment you do not mean what I mean, but in our scenario[1] of memory overcommitment, where memory balloon drivers are used, we configured the memory allocations to fail when there was not enough physical memory, so that the guest application would be able to tell when there is memory pressure. Otherwise, the burden of handling memory pressure is laid purely on the operating system itself.</div><div><br></div><div>[1] <a href="http://www.cs.technion.ac.il/~ladypine/vee18-agmon-ben-yehuda.pdf" style="font-family:'Times New Roman';font-size:medium" target="_blank">Ginseng: Market-Driven Memory Allocation</a><span style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:medium">, Orna Agmon Ben-Yehuda, Eyal Posener, Muli Ben-Yehuda, Assaf Schuster, Ahuva Mu'alem. In proceedings of VEE 2014.</span></div><div> </div></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Sat, May 16, 2015 at 9:14 PM, Elazar Leibovich <span dir="ltr"><<a href="mailto:elazarl@gmail.com" target="_blank">elazarl@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>The question of whether to use a global malloc function, or to use a function pointer is orthogonal to my question.</div><div><br></div><div>My question is, should I support the case of malloc failure. On one hand, it complicates the API significantly, but on the other hand it might be useful for some use cases.</div><div><br></div><div>It's pretty obvious to me that in a modern Linux userspace program, supporting malloc failure does not worth the trouble. But are there other use cases where it's vital?</div><div><br></div><div>Another clarification, my code would never have abort. What I was saying, that the malloc could simply abort current task, if it does not have memory.</div><div><br></div>As a side note, In my experience, it is sometimes useful to use preallocated "memory pools"[0]. Letting the user choose memory allocator is also useful when using it in the kernel, since otherwise the library simply won't compile. See for example protobuf-c which receives an allocator in its functions, <a href="https://github.com/protobuf-c/protobuf-c/blob/master/protobuf-c/protobuf-c.c#L2019" target="_blank">https://github.com/protobuf-c/protobuf-c/blob/master/protobuf-c/protobuf-c.c#L2019</a><div> <br></div><div><div><br></div><div>[0] <a href="http://eli.thegreenplace.net/2008/10/17/memmgr-a-fixed-pool-memory-allocator" target="_blank">http://eli.thegreenplace.net/2008/10/17/memmgr-a-fixed-pool-memory-allocator</a></div></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 15, 2015 at 9:00 PM, Baruch Even <span dir="ltr"><<a href="mailto:baruch@ev-en.org" target="_blank">baruch@ev-en.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>I would question the need to abstract away the memory allocations of your library compared to everything else. If someone cares enough about it he can replace malloc and free completely to use a different allocation scheme.<br><br></div>In most cases I've cared about memory allocations I just wanted none of them at all and only wanted intrusive data structures and just running the system with a fixed memory allocation from the start to the end. It's not always possible in a generic library though..<br><br></div><div>If you are writing a library you should never abort inside it, that would be very annoying to the user. Give him a null and let him crash or handle it as he sees fit.<br></div><div><br></div>Baruch<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Fri, May 15, 2015 at 5:47 PM, Elazar Leibovich <span dir="ltr"><<a href="mailto:elazarl@gmail.com" target="_blank">elazarl@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">I'm writing a small C library, that I want to open source.<div><br></div><div>I want them to be usable for embedded environment, where memory allocation must be controlled.</div><div><br></div><div>Hence, I abstracted away calls to malloc/realloc, and replaced them with</div><div><br></div><div>struct mem_pool {</div><div> void *(*allloc)(void *mem_pool, void *prev_ptr, int size);</div><div>};</div><div><br></div><div>User would implement</div><div><br></div><div>struct my_mem_pool {</div><div> struct mem_pool pool;</div><div> ...</div><div>};</div><div><br></div><div>struct my_mem_pool pool = { { my_alloc_func }, ...);</div><div><br></div><div>I've had to design question I'm interested with:</div><div><br></div><div>1) Should I support both malloc and realloc?</div><div><br></div><div>I think the performance benefits of supporting malloc instead of realloc(NULL) are negligible, and not worth complicating the interface.</div><div><br></div><div>2) Should the memory pool be allowed to fail?</div><div><br></div><div>In typical Linux system, where memory overcommit is allowed, checking malloc return value provides little benefit. But is it the same for embedded system?</div><div><br></div><div>My feeling is, embedded system should predict the memory usage for each input size, and avoid processing input which is too large.</div><div><br></div><div>For example, stack overflow error can never be handled, and one is expected to calculate the longest stack length for any input and make sure he wouldn't overflow.</div><div><br></div><div>So I think it's still reasonable never to report allocation failure, and to expect the memory allocator to raise the relevant abort/panic/exception in such a case.</div><div><br></div><div>But I'll be happy to hear other considerations I missed.</div><div><br></div><div>Thanks,</div></div>
<br></div></div>_______________________________________________<br>
Linux-il mailing list<br>
<a href="mailto:Linux-il@cs.huji.ac.il" target="_blank">Linux-il@cs.huji.ac.il</a><br>
<a href="http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il" target="_blank">http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Linux-il mailing list<br>
<a href="mailto:Linux-il@cs.huji.ac.il" target="_blank">Linux-il@cs.huji.ac.il</a><br>
<a href="http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il" target="_blank">http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div>Orna Agmon Ben-Yehuda.<br><a href="http://ladypine.org" target="_blank">http://ladypine.org</a></div>
</font></span></div>
</blockquote></div><br></div>