© University of Kent - Contact | Feedback | Legal | FOI | Cookies
Benchmark precision and random initial state
Tomas Kalibera, Lubomir Bulej, and Petr Tuma
In Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS), pages 182-196. SCS, March 2005.Abstract
The applications of software benchmarks place an obvious demand on the precision of the benchmark results. An intuitive and frequently employed approach to obtaining precise enough benchmark results is having the benchmark collect a large number of samples that are simply averaged or otherwise statistically processed. We show that this approach ignores an inherent and unavoidable nondeterminism in the initial state of the system that is evaluated, often leading to an implausible estimate of result precision. We proceed by outlining the sources of nondeterminism in a typical system, illustrating the impact of the nondeterminism on selected classes of benchmarks. Finally, we suggest a method for quantitatively assessing the influence of nondeterminism on a benchmark, as well as approach that provides a plausible estimate of result precision in face of the nondeterminism.
Download publication 1000 kbytes (PDF)Bibtex Record
@inproceedings{3143,
author = {Tomas Kalibera and Lubomir Bulej and Petr Tuma},
title = {Benchmark Precision and Random Initial State},
month = {March},
year = {2005},
pages = {182-196},
keywords = {determinacy analysis, Craig interpolants},
note = {},
doi = {},
url = {http://www.cs.kent.ac.uk/pubs/2005/3143},
publication_type = {inproceedings},
submission_id = {21011_1312146091},
other_year = {2005},
ISBN = {1-56555-300-4},
booktitle = {Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS)},
publisher = {SCS},
refereed = {yes},
}