same application code - different behavior with kernels 2.4 vs. 2.6

[Alexander Indenbaum]

Hey,

I'm playing with a Linux embedded box. Some time ago, we (finally)
switched from kernel 2.4.36 to 2.6.18.8. Naturally libc was also
upgraded. What have not changed was the sources of the applications:
Firefox 2.0 running on top of custom X server. GTK/pango/cairo/etc are
also identical in both systems. So far so good, yet we have found out
that the kernel/libc upgrade basically broke the customer's code:
CSS/Javascript timers driven application, running inside the browser,
which started to run much slower, from user experience point of view.
Unfortunately, the customer's user experience is extremely important
;)

Black box analysis showed differences in system behavior, see bellow.
Any idea about what else should be checked are greatly appreciated.
And yeah, the final goal is a happy customer. Please share you
thoughts - bounce some ideas.

Thanks in advance,
~baum


---------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOP
2.4 The system CPU is 5% idle, utilization is between browser and X
server in 60%-40% ratio.
2.6 The system CPU is 80% idle, utilization is between browser and X
server in 60%-40% ratio.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------
STRACE (strace -c -p PID / during 100 secs). NOTE: In case of Firefox,
it is not a really apple to apple comparison,  since only "active
thread" pid was straced.
2.4 kernel - "FAST user experience"
X server
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 90.70   15.259293        1045     14600       204 select
 4.55    0.765248          19     39678     11956 read
 3.59    0.604554          33     18350         2 writev
 0.52    0.087052          62      1415           munmap
 0.46    0.077625           3     29071           gettimeofday
 0.10    0.017304           3      4951      1821 sigreturn
 0.07    0.011463           8      1415           old_mmap
 0.01    0.002002           3       764           setitimer
------ ----------- ----------- --------- --------- ----------------
100.00   16.824541                110244     13983 total

Firefox browser
------ ----------- ----------- --------- --------- ----------------
 47.54    3.022460        1947      1552           select
 19.00    1.207781         272      4446           poll
 12.43    0.789926          41     19280           write
 8.32    0.529035          29     18218        37 readv
 7.37    0.468285         179      2618         2 writev
 1.51    0.095897           5     19428      1253 read
 1.20    0.076559          50      1544           old_mmap
 0.75    0.047951           7      6426           brk
 0.64    0.040452           9      4471           ioctl
 0.58    0.036982          24      1545           munmap
 0.57    0.035924           2     15927           gettimeofday
 0.09    0.005809           5      1156           kill
------ ----------- ----------- --------- --------- ----------------
100.00    6.357061                 96611      1292 total

2.6 kernel - "SLOW user experience"
X server
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 55.00    0.003374           1      5287           writev
 35.18    0.002158           0      8580      2217 select
 9.81    0.000602           0     14486      6008 read
 0.00    0.000000           0     17114           gettimeofday
 0.00    0.000000           0        41           munmap
 0.00    0.000000           0      4454           setitimer
 0.00    0.000000           0      2543      2479 sigreturn
 0.00    0.000000           0        41           mmap2
------ ----------- ----------- --------- --------- ----------------
100.00    0.006134                 52546     10704 total

Firefox browser
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 81.95    0.021973           3      7421           writev
 13.55    0.003634           0     13674      5906 read
 3.68    0.000986           0     13012           select
 0.51    0.000138           1       170           stat64
 0.31    0.000082           1       118           futex
 0.00    0.000000           0        90           write
 0.00    0.000000           0      1789           gettimeofday
 0.00    0.000000           0       249           poll
------ ----------- ----------- --------- --------- ----------------
100.00    0.026813                 36523      5906 total