Benchamark for Beginner Series

1. [Haskell] Stack Setup with pacakge.yaml
2. [Haskell] Write Benchmark with Criterion
3. [Go] Builtin Bechmark with Go

Credit

• This article is based on the Official Document.

• [org-mode] turn off underscore to subscript action on hakyll.

  Please See the footnote.

Go Builtin Benchmark and Test

As a modern development tool, go language is shipped with builtin benchmark tool(module)! And there are some rules to follow. let’s find out.

File Name with surfix “_test.go”

go test -bench=. in the shell (or command) will find any files have the name ends with _test.go to recognize handle it as a test code. For example I’d like make a benchmark code about combinations, I need to make a file with combinations_test.go

sh> head -n 6 combinations_test.go
package main

import "testing"

/* my own module for go lang */
import jeongoon "github.com/jeongoon/go-combinations"

Package name as “main”

as it says:

package main

otherwise cannot find the module automatically by running go test

sh> go test -bench=.
can't load package: package .: found packages main (someCode.go) and in /your/path/to/code

Importing “testing” module

import "testing"

This is the package all we need. simple.

Any Function has prefix “Benchmark”

In your code, there might be some helper function which does not perform benchmark, On the other hand, your benchmark code should start with Benchmark prefix.

func Benchmark_something(b *testing B) {
        // do your benchmark
}

And also note that the argument it will take is b *testing B.

please note that you don’t need to put ‘_’ between Benchmark and something. you only need Benchmark prefix. That’s only personal taste matters.

Issues

Go language has simple and easy rule to follow. however I found some issues about unreliable benchmark time.

Warning: This is not proved but only my opinion. I hope this information will help you get more accurate result out of the benchmark in go language.

warming up code

when you are running and compare the performance between two similar functions do the same job. you might need to put some extra codes which wrap around your real benchmark code.

To test aFunc, you might need to put a warming up code before doing that.

func aFunc() {
        // some codes go here
}

func Benchmark_warmingup_aFunc() {
        someTimes := 10
        for j := 0; j < someTimes; j++ {
                aFunc()
        }
}

func Benchmark_actually_aFunc() {
        for j := 0; j < 10000; j++ {
                aFunc()
        }
}

otherwise the results looks slower on the first run or vice versa.

code after another

I used the same function(aFunc) in the prior example, however when we are comparing other types of codes, we need another warming up code will be required to achieve the even effects as the prior code.

func aFunc() {
        // some codes go here
}

func bFunc() {
        // another implementation goes here
}

// .. snip ..

func Benchmark_actually_bFunc() {
        for j := 0; j < 10000; j++ {
                aFunc()
        }
}

// and warming up again
func Benchmark_warmingup_bFunc() {
        someTimes := 10
        for j := 0; j < someTimes; j++ {
                bFunc()
        }
}

func Benchmark_actually_bFunc() {
        for j := 0; j < 10000; j++ {
                bFunc()
        }
}

outro code for last test

and for some unknown reason, last benchmark does not seems to show correctly measured time. Last one looked like finishing rashly. This is because maybe I have perception already. but if the last code block get affected by the position, we will need to re-arrange them to see the changes.

Or just put another outro code. And it will not harm if there was no effect depending on whether it is located on the last or not, either.

So, I tried to put another cFunc or just another aFunc after testing bFunc block.

func Benchmark_actually_bFunc() {
        for j := 0; j < 10000; j++ {
                bFunc()
        }
}
func Benchmark_outro_aFunc() {
        for j := 0; j < 10; j++ {
                aFunc()
        }

Again, I hope it will help you find the more trustful benchmark result out of go benchmark.

If you’d like to get an example of the benchmark code of mine, please visit my combinations-bench repository.

Org-mode issues

I wrote this article with org-mode in emacs. However if i wrote some word starting with underscore (i.e: _test.go), org-mode interpret as subscript (smaller font under the baseline). #+OPTION: ^:{} will prevents from this behaviour but if you really need subscript you could still use via _{some_scubscript_words} will give me _{some_scubscript_words}, cool. I found this solution¹ on stackoverflow.com

Footnotes