Overview

This a feature request for a tool that allows to compare the results of multiple experiments with the baseline profile. That would allow to validate the results of the experiments:

• Check that all mojos that are expected to be executed are actually executed (i.e., no experiment accidentally excludes a mojo, distorting the total time; or that deliberately excluded mojo is definitely not executed)
• See anomalies and detect invalid experiments. For example, if the first execution starts with empty target/s, but subsequent — with already compiled classes and do not include clean — the compile time in those experiments may be dramatically smaller, which distorts the results. Such issue applies to any executions that use caches.

Manual comparison is tedious and error-prone in case of multiple experiments, especially if the mojos are sorted by execution time.

Reports

Such tool may produce such reports:

Detailed per-mojo comparison for each module

As a web-page:

Module Foo:

A summary of all experiments

As console output/web-page:

* — percents here and elsewhere are mostly random 🙃