<div dir="ltr">The standard deviation can give you an estimation of the minimal running time. (99.xxxx of the samples are within X standard deviations below the average. Pick a high enough xxxx relative to the number of times you'll run the software, and you'll get an estimation of the minimum running time you'll get).<br>
<br>The scalar representing the minimum running time have the obnoxious mis-feature that it doesn't tend to a certain value. So with a small probability you can sample your input today 1m times, and tomorrow yet another 1m times, but still get very different results because of a very rare event. The probability this would happen with the mean or the standard deviation is much much much lower.<br>
<br>But I know almost nothing about statistical analysis, so don't take my word for it, and I'll be glad to hear any corrections.<br><br><div class="gmail_quote">On Sun, Jul 18, 2010 at 1:35 PM, Shlomi Fish <span dir="ltr"><<a href="mailto:shlomif@iglu.org.il">shlomif@iglu.org.il</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im">On Sunday 18 Jul 2010 11:56:33 Elazar Leibovich wrote:<br>
> While you're on to it.<br>
> I expect to read in a benchmark report, the number of time the software was<br>
> executed, the mean running time, and the standard deviation. Running and<br>
> timing it once can hide a pretty large error.<br>
><br>
<br>
</div>Perhaps you're right about that for more complex software. But for Freecell<br>
Solver, I noticed that running it on the virtual console without X running and<br>
while renicing it to the maximal possible priority, yields about the same<br>
runtime every time. This maybe because it is almost purely CPU/memory bound<br>
and does not use the disk much.<br>
<br>
I'm also more interested in the minimal running time than I am in the mean<br>
one.<br>
<div class="im"><br>
Regards,<br>
<br>
Shlomi Fish<br>
<br>
--<br>
-----------------------------------------------------------------<br>
Shlomi Fish <a href="http://www.shlomifish.org/" target="_blank">http://www.shlomifish.org/</a><br>
</div>"Humanity" - Parody of Modern Life - <a href="http://shlom.in/humanity" target="_blank">http://shlom.in/humanity</a><br>
<div><div></div><div class="h5"><br>
God considered inflicting XSLT as the tenth plague of Egypt, but then<br>
decided against it because he thought it would be too evil.<br>
<br>
Please reply to list if it's a mailing list post - <a href="http://shlom.in/reply" target="_blank">http://shlom.in/reply</a> .<br>
</div></div></blockquote></div><br></div>