Benchamark for Beginner Series

1. [Haskell] Stack Setup with pacakge.yaml
2. [Haskell] Write Benchmark with Criterion
3. [Go] Builtin Benchmark with Go

Criterion

criterion is a haskell package to create benchmarks to test speed of your codes.

And I’m going to how to write down the criterion with stack - a haskell project managing programme

stack

stack helps you to create your haskell package. More documents can be found here.

sh> stack new bench-example
Downloading template "new-template" to create project "bench-example" in bench-example/ ...
... snip ...

Selecting the best among 21 snapshots...                                        

* Matches https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/19/3.yaml

Selected resolver: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/19/3.yaml
Initialising configuration using resolver: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/19/3.yaml
Total number of user packages considered: 1                                     
Writing configuration to file: bench-example/stack.yaml                            
All done.                                                                       
/home/myoungjin/.stack/templates/new-template.hsfiles:    6.06 KiB downloaded...

stack downloads some list of latest recommended packages based on a ghc (haskell compiler).

I’m going to use older ghc version because when ghc version is changed you need to build a lot of packages if run it for the first time.

In my case I’m going to stick with lts-16.27. and following file is stack.yaml under your project directory.

# I found the line something like below and commented out
#resolver:
#  url: https://raw.githubusercontent.com/commercialhaskell/stackage-snapshots/master/lts/19/2.yaml
# and add below line
resolver: lts-16.27

In this way, you can keep using your any packages compiled and installed previously. But you don’t have to do if you want to use latest ghc version.

Do Not Touch Your project.cabal

And you will see bench-example.cabal in your directory. stack seems to use cabal to compile the programme or library. However, it is not good idea to edit this file directly because stack will read package.yaml file and automatically update this bench-example.cabal file. So any change you made will be over-written by stack.

You can actually see the message from bench-example.cabal like below.

cabal-version: 1.12

-- This file has been generated from package.yaml by hpack version 0.34.4.
--  
-- see: https://github.com/sol/hpack

package.yaml

Instead, you will need to have a look package.yaml file to apply any changes.

When you open *.cabal file, you will see the some simliar contents with package.yaml So, package.yaml is the file you need to handle.

To make only benchmark programme or just wanted to benchmark on the fly, You need to add a clause called benchmarks:

When Benchmark only

• Firstly, find the tests: clause and copy&paste.

tests:
bench-example-test:
  main:                Spec.hs
  source-dirs:         test
  ghc-options:
  - -threaded
  - -rtsopts
  - -with-rtsopts=-N
  dependencies:
  - haskell-combinations

and copy and paste and modify, please read commments on the right-hand side.

benchmarks:
  bench-example-benchmark:           # any name you would like to use.
    main:               Bench.hs     # this is the file where you write the benchmark code
    source-dirs:        benchmark    # specify directory for benchmark
    ghc-options:
    - -O2                            # I added optimization
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - haskell-combinations
    - criterion                      # criterion is the package for benchmark

one more thing I’d like to mention is that the source-dirs: cannot be shared other app or test or benchmark. because sources-dirs have to have only one Main.hs whose module name is Main.

• and make some example benchmark codes in benchmark directory.

-- file: benchmark/Bench.hs
module Main where

import Criterion
import Criterion.Main (defaultMain)

main :: IO ()
main = do
  defaultMain
  [ bgroup "Some benchmark Group"
    [ bench "Test1" $ nf yourCodeToBenchmark args
    ]
  ]

the prior example is only pseudo code and I’m going to explain in another article.

• finally test on command line. the following code will generate benchmark results.

sh> stack build bench-example:bench:bench-example-benchmark
sh>            # ^ package name
sh>                          # ^ in benchmarks
sh>                                 # ^ benchmakr name (identifier)

Creating Benchmark Executable

• This is similar to making a ‘app’ which goes under executables: clause. Basically, we are using same Bench.hs as a Main.

executables:
  bench-example-benchmark-exe:          # the actual executable name
    main:               Bench.ls        # modulde must be 'Main'
    source-dirs:        benchmark       # we can make executable from prior source file
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - haskell-combinations
    - criterion                         # don't forget the main package.

But please remember you might need to use separate directory for each benchmark if you want to make another type of benchmark is different which desired to execute separately. so /“Different Main/ in Different source directory"

• and emerge it from the command line.

sh> stack build bench-example:exe:bench-example-benchmark-exe
sh>            # ^ it is under the same package
sh>                          # ^ now in __executables__
sh>                                 # ^ benchmakr name (identifier)
sh> stack exec bench-example-benchmark-exe -- -o "report.html"

Why A Benchmark Programme is important?

If tests are to prove that your implementation is right. On the other hand, Benchmarks are journey to find the faster way to your destination.

It could take very long time to reach the destination this is the main reason we need a tool to make it easier.

And Faster doesn’t always need to obey all the rules in theory. So, IMHO, It is more like engineering than science.

That’s why we need objective proof that measure “A” solution saves more time to get the same result. (or less power to get the same thesedays)

Criterion will guide us reasonable results based on your preset parameters one by one. and one of best thing about criterion is that it tries to give as accurate as possible. And also It does give even nice html report as if you pass -o some.html option.

Any Issues?

On my laptop, the results are diffenent from time to time, I vaguely guess that is because I’m using a linux kernel to dedicated to GUI. (something … -zen kernel)

So, I need to keep my laptop status steady during the test. or I needed to change the order of benchmarks to see any affects on the order of executing.

Thank you always. I’ll post about how to write down a example of benchmark code in Go langauge any time soon.