• home
• current students
• graduate
• events

Presenter: Ms. Susan Sim - Computer Science Department, University of Toronto
Supervisor: Dr. R. Nigel Horspool, Chair, Computer Science Department

Date: Mon, March 31, 2003
Time: 13:30:00 - 14:30:00
Place: Engineering Office Wing Building (EOW), Room # 430

ABSTRACT

ABSTRACT:

Benchmarking has been used to compare the performance of a variety of technologies, including computer systems, information retrieval systems, and database management systems. In these and other research areas, benchmarking has caused the science to leap forward. Until now, research disciplines have enjoyed these benefits without a good understanding of how they were achieved. In this talk, I present a process model and a theory of benchmarking to account for these effects. These were developed by examining case histories of existing benchmarks and my own experience with community-wide tool evaluations in software reverse engineering. According to the theory, the tight relationship between a benchmark and the scientific paradigm of a discipline is responsible for the leap forward. A benchmark operationalizes a scientific paradigm; it takes an abstract concept and turns it into a guide for research. Application of this theory will be illustrated using an example from reverse engineering: the C++ Extractor Test Suite (CppETS), a benchmark for comparing fact extractors for the C++ programming language. This talk will conclude with a discussion of how benchmarking can improve the science in software engineering.

Note: Ms. Susan Sim is a candidate for a faculty position in the Department of Computer Science