Skip to content

Commit

Permalink
Establish new benchmark baseline
Browse files Browse the repository at this point in the history
The pathological benchmark runs too fast to be meaningful, and because
it only contains a single path, it doesn't make use of multi-threading
like the other test cases do.

So, let's set up a new baseline by adding more paths to the test case
and running it enough times for it to be within the same order of
magnitude as its immediate neighbors.

  Before:
                          user     system      total        real
  pathological        0.000000   0.000000   0.000000 (  0.002413)
  command-t           0.380000   0.000000   0.380000 (  0.380633)
  chromium (subset)   0.850000   0.010000   0.860000 (  0.441269)
  chromium (whole)    3.400000   0.010000   3.410000 (  1.342450)

  After:
                          user     system      total        real
  pathological        0.110000   0.000000   0.110000 (  0.112132)
  command-t           0.340000   0.000000   0.340000 (  0.340255)
  chromium (subset)   0.800000   0.010000   0.810000 (  0.398784)
  chromium (whole)    3.320000   0.010000   3.330000 (  1.236914)

As you can see there is some flutter between runs (the timings for the
other test cases shouldn't have changed), but I am doing best-of-three
runs for all of these, so that's as close as I can get to scientific.

Signed-off-by: Greg Hurrell <greg@hurrell.net>
  • Loading branch information
wincent committed May 25, 2014
1 parent 94bd7d0 commit 406a16f
Showing 1 changed file with 17 additions and 2 deletions.
19 changes: 17 additions & 2 deletions data/benchmark.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,24 @@
---
tests:
- name: pathological
times: 20
times: 200
paths:
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaa
- aaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
queries:
- a
- aaa
Expand Down

0 comments on commit 406a16f

Please sign in to comment.