New Freecell Solver gcc-4.5.0 vs. LLVM+clang Benchmark

[Nadav Har'El]

On Tue, Feb 01, 2011, Tzafrir Cohen wrote about "Re: New Freecell Solver gcc-4.5.0 vs. LLVM+clang Benchmark":
> On Tue, Feb 01, 2011 at 12:49:00PM +0200, Elazar Leibovich wrote:
> 
> > The program they tested[1] is strictly CPU bound. BTW, standard deviation
> > wouldn't work as well, but it shows (I think) that there's  no such think as
> > "ideal minimal runtime".
> > 
> > [1]
> > static int i = 0, j = 0, k = 0;
> > int main() {
> > int g = 0, inc = 1;
> > for (; g<65536; g++) {
> >   i += inc;
> >   j += inc;
> >   k += inc;
> > }
> > return 0;
> > }
> > 
> > [2] http://www-plan.cs.colorado.edu/diwan/asplos09.pdf
> 
> That program is CPU-bound, but the time it runs is short enough for the
> size of the environment to actually mean something.

Right, I think this example demonstrates Oleg's point, that there's usually
no in-determinism, but rather just ignorance of what else is going on outside
your program.

In this case, this is a program that finishes ridiculously fast - on my
low-end computer, it finishes in 0.3 milliseconds (!). But when you run
this program, things like fork, disk read, system calls, dynamic linking,
and a whole lot of other crap add much more than 0.3 milliseconds of overhead,
and much of that other crap is undeterministic - e.g., if the executable
needs to be read from disk, and something else is using the disk, you notice
a slowdown.

But still - my original point was that if you measure this program's runtime
several times, the minimum, not the average, is often more representative of
the "true" runtime (in this case 0.3 milliseconds). I measured this program
with "time", and one time got 30 milliseconds, several times got 2 milliseconds,
and one time got 1 millisecond. Yes, that minimum of 1 millisecond is closer
to the performance of the program than the average of 1, 2 and 30.

Anyway, you piqued my curiousity, and I'll read that ASPLOS paper :-)

-- 
Nadav Har'El                        |   Wednesday, Feb  2 2011, 28 Shevat 5771
nyh at math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |I saw a book titled "Die Microsoft
http://nadav.harel.org.il           |Windows". Turns out it was in German...