School of Computing

Automated benchmarking and analysis tool

Tomas Kalibera, Jakub Lehotsky, David Majda, Branislav Repcek, Michal Tomcanyi, Antonin Tomecek, Petr Tuma, and Jaroslav Urban

In Proceedings of the 1st International Conference on Performance Evaluation Methodolgies and Tools (VALUETOOLS), pages 182-196. ACM, October 2006 [doi].

Abstract

Benchmarking is an important performance evaluation technique that provides performance data representative of real systems. Such data can be used to verify the results of performance modeling and simulation, or to detect performance changes. Automated benchmarking is an increasingly popular approach to tracking performance changes during software development, which gives developers a timely feedback on their work. In contrast with the advances in modeling and simulation tools, the tools for automated benchmarking are usually being implemented ad-hoc for each project, wasting resources and limiting functionality.We present the result of project BEEN, a generic tool for automated benchmarking in a heterogeneous distributed environment. BEEN automates all steps of a benchmark experiment from software building and deployment through measurement and load monitoring to the evaluation of results. The notable features include separation of measurement from the evaluation and ability to adaptively scale the benchmark experiment based on the evaluation. BEEN has been designed to facilitate automated detection of performance changes during software development (regression benchmarking).

Download publication 171 kbytes (PDF)

Bibtex Record

@inproceedings{3141,
author = {Tomas Kalibera and Jakub Lehotsky and David Majda and Branislav Repcek and Michal Tomcanyi and Antonin Tomecek and Petr Tuma and Jaroslav Urban},
title = {Automated benchmarking and analysis tool},
month = {October},
year = {2006},
pages = {182-196},
keywords = {determinacy analysis, Craig interpolants},
note = {},
doi = {10.1145/1190095.1190101},
url = {http://www.cs.kent.ac.uk/pubs/2006/3141},
    publication_type = {inproceedings},
    submission_id = {8105_1312057129},
    other_year = {2006},
    ISBN = {1-59593-504-5},
    booktitle = {Proceedings of the 1st International Conference on Performance Evaluation Methodolgies and Tools (VALUETOOLS)},
    publisher = {ACM},
    refereed = {yes},
}

School of Computing, University of Kent, Canterbury, Kent, CT2 7NF

Enquiries: +44 (0)1227 824180 or contact us.

Last Updated: 21/03/2014