New Freecell Solver gcc-4.5.0 vs. LLVM+clang Benchmark

[Elazar Leibovich]

On Tue, Feb 1, 2011 at 12:09 PM, Nadav Har'El <nyh at math.technion.ac.il>wrote:

> But if he runs his deterministic application 5 times and gets 5 different
> durations, each duration is composed of the deterministic run-time of
> the application plus a random delay caused by other things on the system.
> The *minimum* of these 5 durations is the one that had the minimum random
> delay, and is thus closest to the "true" run time. Presumably, if one runs
> the application on a machine which is as idle as humanly possible, you'd
> get something close to this minimum.
>
>
See this paper[2] which is referred in the slides.
Maybe the minimal result you're getting is due to the UNIX environment
variables?

We see that something external and orthogonal

to the program, i.e., changing the size (in bytes) of an unused

environment variable, can dramatically (frequently by about

33% and once by almost 300%) change the performance of

our program. This phenomenon occurs because the UNIX

environment is loaded into memory before the call stack.

Thus, changing the UNIX environment size changes the

location of the call stack which in turn affects the alignment

of local variables in various hardware structures.

This simple example demonstrates that computer systems are sensitive: an
> insignificant and seemingly irrelevant change can dramatically affect the
> performance of the

system. As a consequence of this sensitivity, we will find

that different experimental setups will produce different outcomes and thus
> cause measurement bias.


The program they tested[1] is strictly CPU bound. BTW, standard deviation
wouldn't work as well, but it shows (I think) that there's  no such think as
"ideal minimal runtime".

[1]
static int i = 0, j = 0, k = 0;
int main() {
int g = 0, inc = 1;
for (; g<65536; g++) {
  i += inc;
  j += inc;
  k += inc;
}
return 0;
}

[2] http://www-plan.cs.colorado.edu/diwan/asplos09.pdf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20110201/1e0a3b4f/attachment.html>