Consistent way to hide ignored test cases

[Javran]

I noticed that there isn't a consistent way regarding how some test cases are ignored, to name a few:

• https://github.com/exercism/haskell/blob/main/exercises/practice/pangram/test/Tests.hs
• https://github.com/exercism/haskell/blob/main/exercises/practice/alphametics/test/Tests.hs
• there are probably more examples but I'm too lazy to get a compete list unless forced to do so

I'm wondering if you are open to the idea of using some hspec facilities like configSkipPredicate from Config so those ignored cases can be uncommented but still ignored without explicitly specifying it? (This is of course assumming that all test suites are using or can be adapted to use hspec)

A potential strategy, for example, might be marking ignored tests by prefixing its description with Ignored: , and then instruct users to use stack test --ta="--skip=Ignored:" and remove --skip flag when they are up to the challenge.

[sshine]

Hi @Javran, and thanks for contributing! Sorry about the delay. Using configSkipPredicate and prefixing tests that should be skipped by default with Ignored: sounds like a very fine idea. That should mean that appending --ta="--skip=Ignored:" shouldn't be necessary, shouldn't it? Currently we cannot merge new features until we unbreak CI (#952), but you are welcome to work on this addition for the two unit test suites and we can leave it hanging until CI us unbroken.

[Javran]

Sorry, I was thinking about a design choice when I wrote that: the concern is if we override configSkipPredicate, student might have to either remove this override or change test case name to something not matching Ignored: in test code, in order to run ignored tests. HSpec's runner simply apply match filter and remove those skipped, meaning if we skip something by default, it might not be possible to un-skip through command line.

[asarkar]

Would #1075 help in this case?

[IanP96]

Sorry, I was thinking about a design choice when I wrote that: the concern is if we override configSkipPredicate, student might have to either remove this override or change test case name to something not matching Ignored: in test code, in order to run ignored tests. HSpec's runner simply apply match filter and remove those skipped, meaning if we skip something by default, it might not be possible to un-skip through command line.

@sshine it appears that #952 is completed? I noticed some unmerged PRs and thought they might be waiting for #952 given what you said, so just thought I would let you know

[sshine]

Yes, it is. The premise for this issue can be re-assessed without blockers.