Add option for printing the number of QuickCheck tests and shrinks #431
Conversation
|
@jorisdral I'm keen to avoid adding more command-line options. The interface is already quite cluttered. I'd prefer the following behaviour to be implemented unconditionally: firstly show progress from 0% to 100% while testing ( |
|
@jorisdral what do you think about my suggestion above? No promises, but we might be making a release soon; it would be great if this improvement lands before that. |
|
@Bodigrim I think your suggestion makes sense. I'll see if I can adjust the PR soon |
Currently, `tasty` and `tasty-quickcheck` will print a progress percentage, so one can see test progression. However, once a property finds a failure and starts shrinking, then test progression stops. Importantly, it is not clear how the shrinker is progressing, though it would be useful information to show, e.g., to judge whether a test is shrinking too slow, whether the shrinker loops, or whether a shrunk test case hangs. This commit changes the progress bar to print the number of shrinks in case of test failures. Succesful tests will still print the progress percentage.
jorisdral
force-pushed
the
jdral/show-qc-tests-shrinks
branch
from
968bfe4
to
cc442ff
Compare
|
@Bodigrim I've updated the PR. Let me know if this is what you had in mind |
|
|
||
| -- Based on 'QuickCheck.Test.failureSummaryAndReason'. | ||
| showShrinkCount :: QC.State -> Bool -> String | ||
| showShrinkCount st full = count |
There was a problem hiding this comment.
Can we just showShrinkCount st full = concat [...]?
| count = | ||
| concat [ | ||
| show (QC.numSuccessShrinks st) ++ | ||
| concat [ "." ++ show (QC.numTryShrinks st) | showNumTryShrinks ] ++ |
There was a problem hiding this comment.
I spent quite some time deciphering that concat [ xs | c ] is a fancy way to say if c then xs else "". Could we please use the latter form?
There was a problem hiding this comment.
The intermediate output seems to be something like 2.1 shrinks. What is it supposed to mean?
I understand that you adapted QuickCheck.Test.failureSummaryAndReason, which has the same cryptic behavior. Could we just show (QC.numSuccessShrinks st) ++ " shrink" ++ (if ... then "" else "s")?
Currently,
tastyandtasty-quickcheckwill print a progress percentage, so one can see test progression. However, once a property finds a failure and starts shrinking, then test progression stops. Importantly, it is not clear how the shrinker is progressing, though it would be useful information to show, e.g., to judge whether a test is shrinking too slow, whether the shrinker loops, or whether a shrunk test case hangs. This commit adds a new option to enable printing the number of QuickCheck tests (and shrinks on test failure) in addition to the percentage.I know this feature has been suggested before here: #419 (comment)). In my opinion it would be a generally useful feature to include, and it's optional so that it does not use up too much UI space by default. Let me know what you think!