Basic Concepts

Some of the used terminology may not be usual. To avoid confusion, the following defines the basic concepts

experiment

A combination of benchmark suites and executors.

ReBench executes experiments to collect the desired measurements.

benchmark suite

A set of benchmarks which is used to define experiments.

executor

A named set of settings for the executor of a benchmark suite.

Typically, this is one specific executable with a set of startup parameters. The executable will be used to execute benchmarks from a suite.

benchmark

A program to be executed by an executor.

A benchmark can define a number of different variables that can be varied, for instance, to change the input data set, the number of cores to be used, etc.

variable

A dimension of the benchmark that can be varied to influnce execution characteristics

Currently, we have the notion of input sizes, cores, and other variable values. Each of them is varied independently and can potentially be used to enumerate a large number of runs.

run

A concrete execution of a benchmark by a specific executor.

A run is a specific combination of variables. It can be executed multiple times. Each time is referred to as an invocation. One run itself can also execute a benchmark multiple times, which we refer to as iterations of a run.

One run can generate multiple data points.

invocation

The execution of a run. It may execute multiple iterations of a benchmark.

iteration

The execution of a benchmark within an executor invocation. An iteration is expected to generate one data point, possibly including multiple measurements.

data point

A set of measurements generated by an iteration.

measurement

One value for one specific criterion.