FCF 2.0 development in progress...
> > > > > > > >
[News] [C++ Libraries API] [C++ Downloads] [Donate to the project] [Contacts]

outlierThreshold from class fcf::NTest::DurationBasic::BeginOptions

Property: outlierThreshold = -1

Type: double

Class: fcf::NTest::DurationBasic::BeginOptions

Package: fcfTest

File: test.hpp ( link_to_the_declaration_in_the_file )

Available from version: 1.2.10

Threshold for identifying and handling outlier measurements during benchmarking.

The outlierThreshold property defines the statistical limit used to detect outlier iterations during a measurement session. Outliers are measurements that deviate significantly from the established range of the current test.

When an iteration is identified as an outlier:

  • If the ratio of outliers to total iterations is below the threshold, the measurement is discarded, and the benchmark continues.
  • If the ratio reaches or exceeds the threshold, the outlier counter is reset, and the measurement is accepted to prevent a continuous loop of discarding data.

A value of -1 (default) disables this mechanism, meaning all measurements are accepted regardless of how much they deviate.

Example: Filtering Outliers in Benchmarking

Demonstrating how to use the outlierThreshold to automatically discard unstable measurements that deviate from the average performance.

#define FCF_TEST_IMPLEMENTATION #include <fcfTest/test.hpp> #include <chrono> #include <thread> FCF_TEST_DEFINE("Benchmark", "Statistics", "OutlierTest") { // 1. Configure benchmarking options fcf::NTest::Duration::Options options; options.iterations = 100; // 2. Set outlier threshold to 0.20 (20%). // If more than 20% of measurements are outliers, the mechanism resets. options.outlierThreshold = 0.2; // 3. Create the duration object with custom options fcf::NTest::Duration bench(options); // 4. Perform measurement bench([&]() { // Simulate a workload with occasional high-latency spikes (outliers 10%) static int count = 1; count++; if (count % 10 == 0) { // Artificial delay to create an outlier std::this_thread::sleep_for(std::chrono::milliseconds(10)); } else { // Normal workload std::this_thread::sleep_for(std::chrono::microseconds(100)); } }); // 5. Output the results fcf::NTest::log() << "Benchmark finished!" << std::endl; fcf::NTest::log() << "Average duration: " << bench.average() << std::endl; fcf::NTest::log() << "Median duration: " << bench.median() << std::endl; fcf::NTest::log() << bench.histogram().toTable() << std::endl; } int main(int a_argc, char* a_argv[]) { bool error = false; fcf::NTest::cmdRun(a_argc, a_argv, fcf::NTest::CRM_RUN, &error); return error ? 1 : 0; }

Output:

Performing the test: "Benchmark" -> "Statistics" -> "OutlierTest" ... > Benchmark finished! > Average duration: 0.000157794 > Median duration: 0.000155777 > ╔════╦═════════════════════════════════╦═══════╗ > ║ # ║ values ║ count ║ > ╠════╬═════════════════════════════════╬═══════╣ > ║ 1 ║ [0.000`153`707 : 0.000`158`027] ║ 74 ║ > ║ 2 ║ [0.000`158`028 : 0.000`162`348] ║ 23 ║ > ║ 3 ║ [0.000`162`349 : 0.000`166`669] ║ 1 ║ > ║ 4 ║ [0.000`166`670 : 0.000`170`991] ║ 0 ║ > ║ 5 ║ [0.000`170`992 : 0.000`175`312] ║ 0 ║ > ║ 6 ║ [0.000`175`313 : 0.000`179`633] ║ 0 ║ > ║ 7 ║ [0.000`179`634 : 0.000`183`955] ║ 0 ║ > ║ 8 ║ [0.000`183`956 : 0.000`188`276] ║ 1 ║ > ║ 9 ║ [0.000`188`277 : 0.000`192`597] ║ 0 ║ > ║ 10 ║ [0.000`192`598 : 0.000`196`919] ║ 1 ║ > ╚════╩═════════════════════════════════╩═══════╝ > [SUCCESS] Test completed successfully (0.149`542`014 sec) [SUCCESS] All tests were completed. Tests: 1 passed, 0 failed, 0 skipped, 1 total Duration: 0.149`542`014 sec

The outlierThreshold helps ensure that the benchmark results (average and median) are not skewed by rare, non-representative system interruptions or spikes.