New Freecell Solver gcc-4.5.0 vs. LLVM+clang Benchmark

[Elazar Leibovich]

The standard deviation can give you an estimation of the minimal running
time. (99.xxxx of the samples are within X standard deviations below the
average. Pick a high enough xxxx relative to the number of times you'll run
the software, and you'll get an estimation of the minimum running time
you'll get).

The scalar representing the minimum running time have the obnoxious
mis-feature that it doesn't tend to a certain value. So with a small
probability you can sample your input today 1m times, and tomorrow yet
another 1m times, but still get very different results because of a very
rare event. The probability this would happen with the mean or the standard
deviation is much much much lower.

But I know almost nothing about statistical analysis, so don't take my word
for it, and I'll be glad to hear any corrections.

On Sun, Jul 18, 2010 at 1:35 PM, Shlomi Fish <shlomif at iglu.org.il> wrote:

> On Sunday 18 Jul 2010 11:56:33 Elazar Leibovich wrote:
> > While you're on to it.
> > I expect to read in a benchmark report, the number of time the software
> was
> > executed, the mean running time, and the standard deviation. Running and
> > timing it once can hide a pretty large error.
> >
>
> Perhaps you're right about that for more complex software. But for Freecell
> Solver, I noticed that running it on the virtual console without X running
> and
> while renicing it to the maximal possible priority, yields about the same
> runtime every time. This maybe because it is almost purely CPU/memory bound
> and does not use the disk much.
>
> I'm also more interested in the minimal running time than I am in the mean
> one.
>
> Regards,
>
>        Shlomi Fish
>
> --
> -----------------------------------------------------------------
> Shlomi Fish       http://www.shlomifish.org/
> "Humanity" - Parody of Modern Life - http://shlom.in/humanity
>
> God considered inflicting XSLT as the tenth plague of Egypt, but then
> decided against it because he thought it would be too evil.
>
> Please reply to list if it's a mailing list post - http://shlom.in/reply .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20100718/230dacc3/attachment.html>