Python question - first call is slower?

[Baruch Siach]

Hi Nadav,

On Tue, Jul 03, 2012 at 05:02:50PM +0300, Nadav Har'El wrote:
> On Thu, Jun 21, 2012, Nadav Har'El wrote about "Re: Python question - first call is slower?":
> > My example program actually *forked* right before that first call.
> > 
> > It turns out that (at least, this is my understanding now) memory allocation -
> > in Python but also in C - is significantly slower during this first call, but
> >...
> > The short explanation is copy-on-write page faults.
> 
> I have since discovered that there's another reason for slow memory
> allocation after a fork, even if we avoid copy-on-write by filling the
> parent's malloc arena before the fork:
> 
> When the child allocates *new* memory with sbrk(), and later uses it, "minor
> page faults" happen causing the kernel to zero the new pages before the
> program can using it. This zeroing takes time, but is important (for
> security reasons). What worries me more, though, are these minor page
> faults. These happen lazily - only when I access a page, a page fault
> happens, so I am getting one minor page fault per page, and these appear
> to have significant overhead (page table locks, TLB flushes, or I don't
> know what).
> 
> What I'd rather happen in the child is that when sbrk() allocates new
> memory, the kernel immediately maps and zeros all these pages - and
> doesn't wait until they are actually used (which I know they will).
> Do anyone have any idea if this can be done, hopefully without modifying
> the kernel? Is there some madvise() or other trick which can be used to
> ensure that a large number of pages get mapped *now* instead of mapping
> them one after another?

You may try using the kernel hugepage support, either transparent 
(Documentation/vm/transhuge.txt), or non-transparent 
(Documentation/vm/hugetlbpage.txt). When your allocated memory is on a single 
huge page, the kernel must map and initialize it all in a single page fault.  
You also get the benefit of smaller TLB pressure. I'm not sure whether you can 
use huge pages with sbrk() though.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -