same application code - different behavior with kernels 2.4 vs. 2.6

Fri Apr 30 23:33:41 IDT 2010

Alex Hi,

Looks like in 2.4 your apps spent most of the time waiting for data
(select) - and in 2.6 they spend more time doing I/O (writev). I
assume this is network I/O - but it might be disk. If it's network,
check for things that might worsen network I/O: MTU / Csum offloads /
Net card driver + settings / epoll etc etc - you know the drill :)

--Aviv

On Tue, Apr 27, 2010 at 06:17, Alexander Indenbaum
<alexander.indenbaum at gmail.com> wrote:
> Hey,
>
> I'm playing with a Linux embedded box. Some time ago, we (finally)
> switched from kernel 2.4.36 to 2.6.18.8. Naturally libc was also
> upgraded. What have not changed was the sources of the applications:
> Firefox 2.0 running on top of custom X server. GTK/pango/cairo/etc are
> also identical in both systems. So far so good, yet we have found out
> that the kernel/libc upgrade basically broke the customer's code:
> CSS/Javascript timers driven application, running inside the browser,
> which started to run much slower, from user experience point of view.
> Unfortunately, the customer's user experience is extremely important
> ;)
>
> Black box analysis showed differences in system behavior, see bellow.
> Any idea about what else should be checked are greatly appreciated.
> And yeah, the final goal is a happy customer. Please share you
> thoughts - bounce some ideas.
>
> Thanks in advance,
> ~baum
>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
> TOP
> 2.4 The system CPU is 5% idle, utilization is between browser and X
> server in 60%-40% ratio.
> 2.6 The system CPU is 80% idle, utilization is between browser and X
> server in 60%-40% ratio.
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
> STRACE (strace -c -p PID / during 100 secs). NOTE: In case of Firefox,
> it is not a really apple to apple comparison,  since only "active
> thread" pid was straced.
> 2.4 kernel - "FAST user experience"
> X server
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  90.70   15.259293        1045     14600       204 select
>  4.55    0.765248          19     39678     11956 read
>  3.59    0.604554          33     18350         2 writev
>  0.52    0.087052          62      1415           munmap
>  0.46    0.077625           3     29071           gettimeofday
>  0.10    0.017304           3      4951      1821 sigreturn
>  0.07    0.011463           8      1415           old_mmap
>  0.01    0.002002           3       764           setitimer
> ------ ----------- ----------- --------- --------- ----------------
> 100.00   16.824541                110244     13983 total
>
> Firefox browser
> ------ ----------- ----------- --------- --------- ----------------
>  47.54    3.022460        1947      1552           select
>  19.00    1.207781         272      4446           poll
>  12.43    0.789926          41     19280           write
>  8.32    0.529035          29     18218        37 readv
>  7.37    0.468285         179      2618         2 writev
>  1.51    0.095897           5     19428      1253 read
>  1.20    0.076559          50      1544           old_mmap
>  0.75    0.047951           7      6426           brk
>  0.64    0.040452           9      4471           ioctl
>  0.58    0.036982          24      1545           munmap
>  0.57    0.035924           2     15927           gettimeofday
>  0.09    0.005809           5      1156           kill
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    6.357061                 96611      1292 total
>
> 2.6 kernel - "SLOW user experience"
> X server
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  55.00    0.003374           1      5287           writev
>  35.18    0.002158           0      8580      2217 select
>  9.81    0.000602           0     14486      6008 read
>  0.00    0.000000           0     17114           gettimeofday
>  0.00    0.000000           0        41           munmap
>  0.00    0.000000           0      4454           setitimer
>  0.00    0.000000           0      2543      2479 sigreturn
>  0.00    0.000000           0        41           mmap2
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.006134                 52546     10704 total
>
> Firefox browser
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  81.95    0.021973           3      7421           writev
>  13.55    0.003634           0     13674      5906 read
>  3.68    0.000986           0     13012           select
>  0.51    0.000138           1       170           stat64
>  0.31    0.000082           1       118           futex
>  0.00    0.000000           0        90           write
>  0.00    0.000000           0      1789           gettimeofday
>  0.00    0.000000           0       249           poll
> ------ ----------- ----------- --------- --------- ----------------
> 100.00    0.026813                 36523      5906 total
>
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>