Some of the used terminology may not be usual. To avoid confusion, the following defines the basic concepts
ReBench executes experiments to collect the desired measurements.
- benchmark suite
- A set of benchmarks which is used to define experiments.
A named set of settings for the executor of a benchmark suite.
A program to be executed by an executor.
A benchmark can define a number of different variables that can be varied, for instance, to change the input data set, the number of cores to be used, etc.
A dimension of the benchmark that can be varied to influnce execution characteristics
Currently, we have the notion of input sizes, cores, and other variable values. Each of them is varied independently and can potentially be used to enumerate a large number of runs.
A run is a specific combination of variables. It can be executed multiple times. Each time is referred to as an invocation. One run itself can also execute a benchmark multiple times, which we refer to as iterations of a run.
One run can generate multiple data points.
- The execution of a run. It may execute multiple iterations of a benchmark.
- The execution of a benchmark within an executor invocation. An iteration is expected to generate one data point, possibly including multiple measurements.
- data point
- A set of measurements generated by an iteration.
- One value for one specific criterion.