© University of Kent - Contact | Feedback | Legal | FOI | Cookies
Precise regression benchmarking with random effects: Improving Mono benchmark results
Tomas Kalibera and Petr Tuma
In Proceedings of the 3rd European Performance Engineering Workshop (EPEW), volume 4054 of Lecture Notes in Computer Science, pages 182-196. Springer, June 2006 [doi].Abstract
Benchmarking as a method of assessing software performance is known to suffer from random fluctuations that distort the observed performance. In this paper, we focus on the fluctuations caused by compilation. We show that the design of a benchmarking experiment must reflect the existence of the fluctuations if the performance observed during the experiment is to be representative of reality.
We present a new statistical model of a benchmark experiment that reflects the presence of the fluctuations in compilation, execution and measurement. The model describes the observed performance and makes it possible to calculate the optimum dimensions of the experiment that yield the best precision within a given amount of time.
Using a variety of benchmarks, we evaluate the model within the context of regression benchmarking. We show that the model significantly decreases the number of erroneously detected performance changes in regression benchmarking.
Download publication 151 kbytes (PDF)Bibtex Record
@inproceedings{3140,
author = {Tomas Kalibera and Petr Tuma},
title = {Precise Regression Benchmarking with Random Effects: Improving {Mono} Benchmark Results},
month = {June},
year = {2006},
pages = {182-196},
keywords = {determinacy analysis, Craig interpolants},
note = {},
doi = {10.1007/11777830_5},
url = {http://www.cs.kent.ac.uk/pubs/2006/3140},
publication_type = {inproceedings},
submission_id = {7810_1312056476},
other_year = {2006},
ISBN = {3-540-35362-3},
booktitle = {Proceedings of the 3rd European Performance Engineering Workshop (EPEW)},
volume = {4054},
series = {Lecture Notes in Computer Science},
publisher = {Springer},
refereed = {yes},
}