Usage

ReBench is a command-line tool. In the following, we will discuss its usage. The complete set of options can be displayed with the --help argument:

$ rebench --help
Usage: rebench [options] <config> [exp_name] [e:$]* [s:$]* [m:$]*

Argument:
  config    required argument, file containing the experiment to be executed
  exp_name  optional argument, the name of an experiment definition
            from the config file
            If not provided, the configured default_experiment is used.
            If 'all' is given, all experiments will be executed.

  e:$       filter experiments to only include the named executor, example: e:EXEC1 e:EXEC3
  s:$       filter experiments to only include the named suite and possibly benchmark
              example: s:Suite1 s:*:Bench3
  t:$       filter experiments to only include the given tags, example: t:machine1 t:tagFoo
...

Configuration files provide the setup for experiments by defining benchmarks, benchmark suites, their parameters, and executors to execute them.

Basic Execution: Run Experiments

ReBench takes a given configuration file, executes the experiments and stores the measurement results into the configured data file. Assuming a basic configuration as seen in the README, the following command line will execute all experiments and store the results in the example.data file:

$ rebench example.conf

This basic execution can be customized in various ways as explained below.

Partial Execution: Run Some of the Experiments

Instead of executing the configured experiments, we can ask ReBench to only execute a subset of them, a specific experiment, only selected executors, suites, and benchmarks.

The configuration file allows us to select a default_experiment. But we can override this setting with the exp_name parameter. Thus, the following will execute only the Example experiment:

$ rebench example.conf Example

We can further restrict what is executed. To only execute the MyBin1 executor, we use:

$ rebench example.conf Example e:MyBin1

To further limit the execution, we can also select a specific benchmark from a suite:

$ rebench example.conf Example e:MyBin1 s:ExampleSuite:Bench1

The filters are applied on the set of runs identified by a configuration and the chosen experiments. Thus, the above says to execute only MyBin1, and no other executor. For the resulting runs, we also want to execute only Bench1 in the ExampleSuite. If we list additional executors, they are all considered. Similarly, naming more benchmarks will include them all.

Further Options

ReBench supports a range of other options to control execution.

Quick Runs, Iterations, Invocations, Building

The configuration uses the notion of iteration and invocation to define how often an executor is started (invocation) and how many times a benchmark is executed in the same executor execution (iteration).

We can override this setting with the following parameters:

-in INVOCATIONS, --invocations INVOCATIONS
                 The number of times an executor is started to execute a run.
-it ITERATIONS, --iterations ITERATIONS
                 The number of times a benchmark is to be executed
                 within an executor execution.

-q, --quick      Execute quickly. Identical with --iterations=1 --invocations=1

--setup-only     Build all executors and suites, and run one benchmark for each executor.
                 This ensures executors and suites are built.
                 It Implies --iterations=1 --invocations=1.

-B, --without-building
                 Disables execution of build commands for executors and suites.

Discarding Data, Rerunning Experiments, and Faulty Runs

ReBench's normal execution mode will assume that it should accumulate all data until a complete data set is reached. This means, we can interrupt execution at any point and continue later and ReBench will continue where it left off.

Some times, we may want to update some experiments and discard old data:

-c, --clean   Discard old data from the data file (configured in the run description).
-r, --rerun   Rerun selected experiments, and discard old data from data file.

Runs may fail for a variety of reasons. A benchmark might be buggy, the executor may be faulty or unavailable, or the run reaches the max_invocation_time. To include all data generated for the benchmark until it failed, we can use the -f option:

-f, --faulty  Include results of faulty or failing runs

Execution Order

We may care for a different order for the benchmark execution. This could either be to get a quicker impression of the performance results, or possibly to account for the complexity of benchmarking and ensure that the order does not influence results.

For this purpose we use schedulers to determine the execution order.

-s SCHEDULER, --scheduler=SCHEDULER
                        execution order of benchmarks: batch, round-robin,
                        random [default: batch]

Prevent Execution to Verify Configuration

To check whether a configuration is correct, it can be useful to avoid execution altogether. For such testing or dry run, we have the following options:

-E, --no-execution    Disables execution. It allows to verify the
                      configuration file and other parameters.
-p, --execution-plan  Print execution plan. This outputs all executions
                      that would be performed, without executing them.

Continuous Performance Tracking

ReBench supports Codespeed and ReBenchDB as platforms for continuous performance tracking. To report data to such a setup, the configuration needs to have the corresponding details.

Additionally, Codespeed and ReBenchDB need details on the concrete execution:

--commit-id COMMIT_ID     MANDATORY: when Codespeed reporting is  used, the
                          commit-id has to be specified.

--environment ENVIRONMENT MANDATORY: name the machine on which the results are
                          obtained.

--branch BRANCH           The branch for which the results have to be recorded,
                          i.e., to which the commit belongs. Default: HEAD

--executable EXECUTABLE   The executable name given to Codespeed. Default: The
                          name used for the executor.

--project PROJECT         The project name given to Codespeed. Default: Value
                          given in the config file.

-I, --disable-inc-report  Creates a final report at the end instead of reporting
                          incrementally.

-R, --disable-data-reporting Override the configuration and disable any reporting
                          to Codespeed and ReBenchDB.

--git-repo GIT_REPO       Path to the git repository that contains the source of this experiment.
                          This is useful when the experiment is run from a different path
                          without the `.git` folder.