Type: double
Class: fcf::NTest::DurationBasic::BeginOptions
Package: fcfTest
File:
test.hpp
( link_to_the_declaration_in_the_file )
Available from version: 1.2.10
Threshold for identifying and handling outlier measurements during benchmarking.
The outlierThreshold property defines the statistical limit used to detect outlier iterations during a measurement session. Outliers are measurements that deviate significantly from the established range of the current test.
When an iteration is identified as an outlier:
- If the ratio of outliers to total iterations is below the threshold, the measurement is discarded, and the benchmark continues.
- If the ratio reaches or exceeds the threshold, the outlier counter is reset, and the measurement is accepted to prevent a continuous loop of discarding data.
A value of -1 (default) disables this mechanism, meaning all measurements are accepted regardless of how much they deviate.
Example: Filtering Outliers in Benchmarking
Demonstrating how to use the outlierThreshold to automatically discard unstable measurements that deviate from the average performance.
#define FCF_TEST_IMPLEMENTATION
#include <fcfTest/test.hpp>
#include <chrono>
#include <thread>
FCF_TEST_DEFINE("Benchmark", "Statistics", "OutlierTest") {
// 1. Configure benchmarking options
fcf::NTest::Duration::Options options;
options.iterations = 100;
// 2. Set outlier threshold to 0.20 (20%).
// If more than 20% of measurements are outliers, the mechanism resets.
options.outlierThreshold = 0.2;
// 3. Create the duration object with custom options
fcf::NTest::Duration bench(options);
// 4. Perform measurement
bench([&]() {
// Simulate a workload with occasional high-latency spikes (outliers 10%)
static int count = 1;
count++;
if (count % 10 == 0) {
// Artificial delay to create an outlier
std::this_thread::sleep_for(std::chrono::milliseconds(10));
} else {
// Normal workload
std::this_thread::sleep_for(std::chrono::microseconds(100));
}
});
// 5. Output the results
fcf::NTest::log() << "Benchmark finished!" << std::endl;
fcf::NTest::log() << "Average duration: " << bench.average() << std::endl;
fcf::NTest::log() << "Median duration: " << bench.median() << std::endl;
fcf::NTest::log() << bench.histogram().toTable() << std::endl;
}
int main(int a_argc, char* a_argv[]) {
bool error = false;
fcf::NTest::cmdRun(a_argc, a_argv, fcf::NTest::CRM_RUN, &error);
return error ? 1 : 0;
}
Output:
Performing the test: "Benchmark" -> "Statistics" -> "OutlierTest" ...
> Benchmark finished!
> Average duration: 0.000157794
> Median duration: 0.000155777
> ╔════╦═════════════════════════════════╦═══════╗
> ║ # ║ values ║ count ║
> ╠════╬═════════════════════════════════╬═══════╣
> ║ 1 ║ [0.000`153`707 : 0.000`158`027] ║ 74 ║
> ║ 2 ║ [0.000`158`028 : 0.000`162`348] ║ 23 ║
> ║ 3 ║ [0.000`162`349 : 0.000`166`669] ║ 1 ║
> ║ 4 ║ [0.000`166`670 : 0.000`170`991] ║ 0 ║
> ║ 5 ║ [0.000`170`992 : 0.000`175`312] ║ 0 ║
> ║ 6 ║ [0.000`175`313 : 0.000`179`633] ║ 0 ║
> ║ 7 ║ [0.000`179`634 : 0.000`183`955] ║ 0 ║
> ║ 8 ║ [0.000`183`956 : 0.000`188`276] ║ 1 ║
> ║ 9 ║ [0.000`188`277 : 0.000`192`597] ║ 0 ║
> ║ 10 ║ [0.000`192`598 : 0.000`196`919] ║ 1 ║
> ╚════╩═════════════════════════════════╩═══════╝
>
[SUCCESS] Test completed successfully (0.149`542`014 sec)
[SUCCESS] All tests were completed.
Tests: 1 passed, 0 failed, 0 skipped, 1 total
Duration: 0.149`542`014 sec
The outlierThreshold helps ensure that the benchmark results (average and median) are not skewed by rare, non-representative system interruptions or spikes.