I often have talks with my colleagues about performance of this or that code snippet. These talks often grow up into a discussion on what solution works faster. In this situation, it’s better to create a simple benchmark than to have a long philosophical talk about performance. Unfortunately, many developers don’t know how to run benchmarks correctly. And as a result their test can be absolutely unreliable. That is why I wanted to discuss the most important things that should be considered while creating the proper C# benchmark.
Release mode without debugging
One of the most often mistakes made while running performance tests is running the application in debug mode. Results you get in this way may be absolutely different and won’t be true to facts. The matter is that, in debug build, C# compiler adds a lot of IL commands that can influence performance. Besides, it’s better to run the application without debugging (Ctrl+F5 from Visual Studio and even better from the console). If you run the application in debug mode (F5 from Visual Studio) then the debug code will spoil the results of your benchmark. You can find a more detailed description of this problem by Eric Lippert.
Warm up processor cache
Your test should definitely run on the warmed up cache. We need to run the test with no load for several times. Some developers run tests with no load only once or twice, but you need to run the test for 10-15 times to make cache really warmed up. It’s better to warm up cache till variation of time measurements from test to test are less than some small per cent. Benchmark results on the cold cache can show much more time than it’s really necessary.
Benchmarking on a single processor
You should also remember that we live in the multiprocessor world and every processor has its own cache. To benchmark on a single processor you need to set the corresponding ProcessorAffinity-mask:
Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(1);
Stopwatch, not DateTime
It’s a good practice to measure application work time using Stopwatch. You can get more information about this from Eric Lippert or on dotnetperls.
High priority
Don’t forget that there are tens of other applications running on your computer. It’s better to shut down the most resource consuming apps while running the test. And it’s even better to set high priority for the process and thread in your application:
Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High;
Thread.CurrentThread.Priority = ThreadPriority.Highest;
Garbage Collection
Don’t forget about the garbage collector that can suddenly run when you don’t expect it at all. And if you need to consider collecting garbage of the objects from the current test, collecting garbage of the objects from the previous test is not required at all. It’s better to call GC.Collect() before benchmarking. And it’s even better to call it twice to make all root objects convert to second generation. It won’t get worse to wait till all finalizers (if any) complete their work (GC.WaitForPendingFinalizers() will help).
Running a target benchmark for several times
A single benchmark can get unreliable result by force of circumstances (for example, due to unexpected activity of the other application). That is why to make the test accurate it’s necessary to benchmark (after warming up cache) for several times and get an average result. It’s also useful to look at the variation of values for different tests to estimate benchmark reliability.
Number of iterations should be a prime number
If the number of iterations in a cycle has divisors, JIT can apply Loop unwinding-optimization. Results of some methods can be affected. It’s good to set number of iteration to a prime number.
________________________________________
BenchmarkDotNet
I listed some important moments that you should remember when creating a benchmark. I like benchmarking. But it’s quite tiresome to make reliable benchmarks in every new application from scratch every time. Besides, I want to get good-looking results, but I don’t want to mess about formatting every time. That is why I created a simple project – BenchmarkDotNet that is available on GitHub. It can conveniently estimate performance of the definite code snippets, it’s just necessary to describe target methods. I hope the project will help other C# developers estimate performance of their code. If you can add something else that should be considered when measuring time I will be glad to get pull requests.
