Benchmarking: High-Precision Performance Measurement

In modern software development, understanding the performance characteristics of your code is just as important as verifying its correctness. fcfTest provides a professional-grade benchmarking utility via the fcf::NTest::Duration class, allowing you to transform standard unit tests into powerful performance verification tools.

Measuring Single Operations

There are two primary ways to measure code execution. Depending on whether you want to measure a single event or a repeated workload, you can use either manual control or the convenient call operator.

Method 1: Manual Control

If you need to wrap a specific section of a test, use the begin and end methods. This is useful when the setup/teardown logic of your test should not be included in the timing.

#define FCF_TEST_IMPLEMENTATION #include <fcfTest/test.hpp> #include <thread> #include <chrono> FCF_TEST_DEFINE("Bench", "Manual", "SingleOp") { // 1. Create a duration object fcf::NTest::Duration bench; // 2. Start the timer bench.begin(); // Perform the operation to be measured std::this_thread::sleep_for(std::chrono::milliseconds(10)); // 3. Stop the timer bench.end(); fcf::NTest::log() << "Measured time: " << bench.duration() << std::endl; } int main(int a_argc, char* a_argv[]) { bool error = false; fcf::NTest::cmdRun(a_argc, a_argv, fcf::NTest::CRM_RUN, &error); return error ? 1 : 0; }

Terminal output

Performing the test: "Bench" -> "Manual" -> "SingleOp" ... > Measured time: 0.010`130`088 [SUCCESS] Test completed successfully (0.010`168`749 sec) [SUCCESS] All tests were completed. Tests: 1 passed, 0 failed, 0 skipped, 1 total Duration: 0.010`168`749 sec

Method 2: Automated Iterations (The Recommended Way)

For statistically significant results, you should run the operation many times. The operator() overload automates this process by executing a provided lambda function for a specified number of iterations.

#define FCF_TEST_IMPLEMENTATION #include <fcfTest/test.hpp> #include <vector> #include <algorithm> FCF_TEST_DEFINE("Bench", "Automated", "LambdaRun") { // 1. Initialize with 1000 iterations fcf::NTest::Duration bench(1000); // 2. Pass the workload as a lambda. // The framework handles the loop, warmup, and timing. bench([]() { std::vector<int> v = {5, 2, 9, 1}; std::sort(v.begin(), v.end()); }); fcf::NTest::log() << "Avg: " << bench.average() << std::endl; fcf::NTest::log() << "Total: " << bench.duration() << std::endl; } int main(int a_argc, char* a_argv[]) { bool error = false; fcf::NTest::cmdRun(a_argc, a_argv, fcf::NTest::CRM_RUN, &error); return error ? 1 : 0; }

Terminal output

Performing the test: "Bench" -> "Automated" -> "LambdaRun" ... > Avg: 0.000000373 > Total: 0.000373003 [SUCCESS] Test completed successfully (0.000`947`775 sec) [SUCCESS] All tests were completed. Tests: 1 passed, 0 failed, 0 skipped, 1 total Duration: 0.000`947`775 sec

Advanced: Multi-Level Benchmarking

Sometimes, you need to understand the hierarchy of performance. For example, you might want to measure the total time of a test, but also isolate the time spent in a specific sub-component or a deep loop within that test.

fcfTest supports Measurement Levels. Each fcf::NTest::Duration object can manage multiple independent timing layers (levels). You can start a measurement at level 0, and then start a nested measurement at level 1.

How it works

Level 0 (Global/Base): Usually represents the entire test or the main loop.
Level 1, 2, ... (Nested): Represents specific segments within the base level.

This allows you to distinguish between the overhead of the test framework itself and the actual execution of your code.

#define FCF_TEST_IMPLEMENTATION #include <fcfTest/test.hpp> #include <thread> #include <chrono> FCF_TEST_DEFINE("Bench", "Nested", "MultiLevel") { fcf::NTest::Duration bench(100); // 1. Start the base measurement (Level 0) bench.begin(0); // Simulate some setup work outside the target loop std::this_thread::sleep_for(std::chrono::milliseconds(5)); // 2. Start a nested measurement (Level 1) for the inner loop fcf::NTest::Duration::Options options; options.iterations = 10; bench(options, 1 /*level*/, [](){ std::this_thread::sleep_for(std::chrono::milliseconds(1)); }); // 3. End the base measurement bench.end(0); // 4. Report both levels fcf::NTest::log() << "Total (L0): " << bench.duration(0) << std::endl; fcf::NTest::log() << "Inner (L1): " << bench.duration(1) << std::endl; FCF_TEST(true); } int main(int a_argc, char* a_argv[]) { bool error = false; fcf::NTest::cmdRun(a_argc, a_argv, fcf::NTest::CRM_RUN, &error); return error ? 1 : 0; }

Terminal output:

Performing the test: "Bench" -> "Nested" -> "MultiLevel" ... > Total (L0): 0.015960201 > Inner (L1): 0.010894881 [SUCCESS] Test completed successfully (0.016`070`914 sec) [SUCCESS] All tests were completed. Tests: 1 passed, 0 failed, 0 skipped, 1 total Duration: 0.016`070`914 sec

Statistical Depth: Histograms

A single average value can be misleading. A performance spike (outlier) might be hidden by a low average, or a high variance might make the average non-representative. To solve this, fcfTest integrates a Histogram directly into the fcf::NTest::Duration object.

Every time an iteration is recorded, the duration is placed into a bin within the histogram. This allows you to retrieve not just the mean, but the median, min, max, and a full visual distribution.

#define FCF_TEST_IMPLEMENTATION #include <fcfTest/test.hpp> #include <thread> #include <chrono> #include <random> FCF_TEST_DEFINE("Stats", "Histogram", "Analysis") { // 1. Configure: 100 iterations, 30 bins for distribution fcf::NTest::Duration::Options opts(100, 30); fcf::NTest::Duration bench(opts); // 2. Execute a workload with varying execution times to see a distribution bench([]() { static std::mt19937 gen(42); std::uniform_int_distribution<int> dist(1, 30); std::this_thread::sleep_for(std::chrono::milliseconds(dist(gen))); }); // 3. Log statistical results fcf::NTest::log() << "--- Statistical Report ---" << std::endl; fcf::NTest::log() << "Median: " << bench.median() << std::endl; fcf::NTest::log() << "Min: " << bench.min() << std::endl; fcf::NTest::log() << "Max: " << bench.max() << std::endl; // 4. Visualize the distribution via ASCII Bar Chart fcf::NTest::log() << "\nDistribution Chart:\n" << bench.histogram().toBarChart() << std::endl; fcf::NTest::log() << "\nDistribution Table:\n" << bench.histogram().toTable() << std::endl; } int main(int a_argc, char* a_argv[]) { bool error = false; fcf::NTest::cmdRun(a_argc, a_argv, fcf::NTest::CRM_RUN, &error); return error ? 1 : 0; }

Terminal output

Performing the test: "Stats" -> "Histogram" -> "Analysis" ... > --- Statistical Report --- > Median: 0.013658538 > Min: 0.001065266 > Max: 0.030126662 > > Distribution Chart: > | > | | > || | | | > || | | | | > || | | | | > || ||| | | | || | || > ||| ||| | ||| |||| | | | || > ||| |||||| |||||||||||| || || > |||||||||||||||||||||||||| ||| > |||||||||||||||||||||||||| ||| > ══════════════════════════════ > OX (value): [0.001`065`266 : 0.030`126`662]; Step: 968713.20 > OY (count): [0 : 8]; Step: 0.80 > > > Distribution Table: > ╔════╦═════════════════════════════════╦═══════╗ > ║ # ║ values ║ count ║ > ╠════╬═════════════════════════════════╬═══════╣ > ║ 1 ║ [0.001`065`266 : 0.002`033`978] ║ 7 ║ > ║ 2 ║ [0.002`033`979 : 0.003`002`691] ║ 6 ║ > ║ 3 ║ [0.003`002`692 : 0.003`971`404] ║ 3 ║ > ║ 4 ║ [0.003`971`405 : 0.004`940`117] ║ 1 ║ > ║ 5 ║ [0.004`940`118 : 0.005`908`831] ║ 4 ║ > ║ 6 ║ [0.005`908`832 : 0.006`877`544] ║ 8 ║ > ║ 7 ║ [0.006`877`545 : 0.007`846`257] ║ 4 ║ > ║ 8 ║ [0.007`846`258 : 0.008`814`970] ║ 2 ║ > ║ 9 ║ [0.008`814`971 : 0.009`783`684] ║ 2 ║ > ║ 10 ║ [0.009`783`685 : 0.010`752`397] ║ 3 ║ > ║ 11 ║ [0.010`752`398 : 0.011`721`110] ║ 1 ║ > ║ 12 ║ [0.011`721`111 : 0.012`689`823] ║ 6 ║ > ║ 13 ║ [0.012`689`824 : 0.013`658`537] ║ 3 ║ > ║ 14 ║ [0.013`658`538 : 0.014`627`250] ║ 4 ║ > ║ 15 ║ [0.014`627`251 : 0.015`595`963] ║ 2 ║ > ║ 16 ║ [0.015`595`964 : 0.016`564`676] ║ 4 ║ > ║ 17 ║ [0.016`564`677 : 0.017`533`389] ║ 3 ║ > ║ 18 ║ [0.017`533`390 : 0.018`502`103] ║ 4 ║ > ║ 19 ║ [0.018`502`104 : 0.019`470`816] ║ 6 ║ > ║ 20 ║ [0.019`470`817 : 0.020`439`529] ║ 2 ║ > ║ 21 ║ [0.020`439`530 : 0.021`408`242] ║ 4 ║ > ║ 22 ║ [0.021`408`243 : 0.022`376`956] ║ 2 ║ > ║ 23 ║ [0.022`376`957 : 0.023`345`669] ║ 3 ║ > ║ 24 ║ [0.023`345`670 : 0.024`314`382] ║ 1 ║ > ║ 25 ║ [0.024`314`383 : 0.025`283`095] ║ 3 ║ > ║ 26 ║ [0.025`283`096 : 0.026`251`809] ║ 2 ║ > ║ 27 ║ [0.026`251`810 : 0.027`220`522] ║ 0 ║ > ║ 28 ║ [0.027`220`523 : 0.028`189`235] ║ 1 ║ > ║ 29 ║ [0.028`189`236 : 0.029`157`948] ║ 5 ║ > ║ 30 ║ [0.029`157`949 : 0.030`126`662] ║ 4 ║ > ╚════╩═════════════════════════════════╩═══════╝ > [SUCCESS] Test completed successfully (1.399`777`530 sec) [SUCCESS] All tests were completed. Tests: 1 passed, 0 failed, 0 skipped, 1 total Duration: 1.399`777`530 sec

Summary of Statistical Methods

Method	Description
duration()	The total execution duration.
average()	The arithmetic mean of all recorded iterations.
median()	The middle value, providing a robust measure against outliers.
min() / max()	The absolute boundaries of the recorded execution times.
histogram()	Returns the full fcf::NTest::HistogramBasic object for advanced analysis.