Actually support inconclusive tests. Stop disabling a popular workflow!
I and many others use inconclusive to be able to sketch/plan a specification. Using a template I create a ton of small specific tests classes and methods that all call Assert.Inconclusive.
Then I(Or another developer!) start actually implementing the tests and the backend code one test at a time. As soon as one test passes I can move on to the next inconclusive test. Once they all pass I am done.
This is my favorite workflow, and I am completely unable to use it with NCrunch. Either I get no indication of what work I have left to do(successful setting), or ncrunch says that I have a ton of failing tests(failure setting). Both of these results are simply incorrect. I have zero failing tests. But I do have a number of Inconclusive tests.
Considering tests that are inconclusive as either Sucessful or Failed does NOT make things simpler. It is a lie that makes things much much harder by disabling the workflows that inconclusive was created to support!
Albert Weinert commented
It even worse, even Skipped Test (at Runtime because of missing preconditions, or whatever) will be marked as inconclusive.
I can live with the ignoring of inconclusive tests, but i would prefer als have the test display with no error in the test runner (marked as inconclusive).
But a explicit skipped test is not inconclusive, it skipped.
NCrunch must not decide was an inconclusive test, the user has to do this.
I've never run in an inconclusive test by accident because something is not working, i run in that with MSpec (which is intentional) and now with Xunit (also intentional in that case).
Being afraid of updating the UI for that is IMHO a somewhat lame excuse.
I can't imagine why it should be confusing to display skipped and inconclusive tests.
Alan Hemmings commented
Hi Remco & ncruch fans!
I'd love to put forward the following argument "for" keeping this feature request open for voting and further debate:
re: what is an inconclusive test :
A: Anywhere where a test exit's via Assert.Inconclusive(); and nowhere else. Test either passes, fails or exits with Inconclusive. I'm pretty certain there's no ambiguity here as this was very clearly defined a long time ago by NUnit.
re: is it a test that is just stubbed : A: NO (same as below)
re: is it a test that hasn't been implemented. A: NO Tthat's a passing test according to Nunit. No ambiguity. If a user opts to take advantage of that, and call Assert.Inconclusive() and have the system report (n) tests are now inconclusive, or that a particular line of code is covered by an inconclusive test, then that's the developer's prerogative, and we've been doing this succesfully for many years with NUnit.
re: is it a malfunctioning test : A : NO There's no such thing as a malfunctioning test, you just made that up. :)
re: is it a test that doesn't apply: NO : Same as above. A developer is quite entitled to do `if !SystemReady() Assert.Inconclusive();` Sure the result might be that at any point NCrunch shows that the code the developer is busy looking at or working on just randomly happens to be covered by an inconclusive test. That's a real use case (imho) and of course, you would work hard to change your tests dependancies to depend on mocks instead of some flaky component to make the test more reliable. but until you've done that, inconclusive represents reality.
re: as soon as we introduce a third state, suddenly many of the underlying concepts that Ncrunch UX ..no longer make sense
mmm, I will trust you that what you are saying is true, however, just saying it, doesn't help us understand. I'm pretty certain the UX would work just fine and I can't imagine (I mean this in a good way, I am actually trying to understand the scenario's you have eluded to. A third state is just an additional colour and state that the developer is already educated about and understands. From our perspective this does not appear complex?
re : The corner spinner that gives a blanket aggregation of ALL the available results and state of the engine
requires one additional configuration setting: Inconclusive tests are/converted to : [ignored] [pass] [Fail]
In my case, with my workflow, same as the OP's is to set Inconclusive to Pass, which (imho) can be the default setting.
re: test window with set of filters
Add one more filter: hide/show inconclusive. exactly the same as the other filters.
BTW: the UX for the filter buttons needs to show whether they are currently selected or not. Seperate issue, but clicking them toggles them, but I have no way of seeing what state the toggle button is currently in.
re: global tests that operate heavily
this seems straight forward : Inconclusive = pass. All logic should work?
set Inconclusive = pass. All logic (and stats) should work correctly. No different to an empty test that returns true anyway even with the current Ncrunch settings. You're no worse off, but you are better off if the workflow helps you. If you don't like this workflow, you'll not be using Assert.Inconclusive anywhere in your code and you're no worse off either.
If you do use Assert.Inconclusive, but for a different workflow, not the same as the OP's then you're absolutely going to want to be able to see which tests are inconclusive.
RE: data exports
Need to see an example from you of where this would be problematic. Still don't see a problem?
Please re-open this option for voting and further debate. I believe the feature will help developers use NCrunch more productively because without this feature we have to swap between ncrunch and resharper and-or other test runner, and this dramatically impacts the overall experience of using ncrunch since I'm now watching two windows full of mostly duplicated output, with only ncrunch being updated in real time.
re: leave the majority of users confused.
I'm not convinced this is the case. Leave this open for voting and further debate. If the users don't call Assert.Inconclusive they won't be confused. Right now not making a decision IS making a decision, and (imho) that decision is sub-optimal.
Your default decision (implicit in the current design) is that Assert.Inconclusive is a combination of PASS and Hidden. (I may have this slightly wrong, but the point I'm making is that you've already made a decision to handle / hide Inconclusives.
Unfortunately, I can't see this ever being implemented in NCrunch. Respecting that this may disappoint a few people, I'll do my best to explain my reasoning here.
The entirety of NCrunch; its backend system, its UX, its workflow, was built on the idea of a test having a binary result. Either a test fails and it requires your attention, or it passes and it does not.
As soon as we introduce a third state, suddenly many of the underlying concepts that the NCrunch UX and workflow were designed on no longer make sense.
What is an inconclusive test? Is it a test that is just stubbed and hasn't been implemented? Is it a malfunctioning integration test? Is it a test that doesn't apply? Is it a test where the result doesn't make sense? The problem with inconclusive tests is that the answer to these questions is often different. In regards to workflow and results reporting, an inconclusive state only has meaning when there is an understanding of how the user wants to interpret it. People use inconclusive results for so many different things that it would be impossible for NCrunch to make sense of these results without a whole new range of configuration settings.
It's easier to understand this problem when you start to look at various components of the NCrunch UI:
- The coverage markers which change colour based on the aggregated results of their covering tests
- The corner spinner that gives a blanket aggregation of ALL the available results and state of the engine
- The progress bar that also aggregates everything with timescales
- The Tests Window with its set of filters, grouping system and status reporting bar
- The global test context commands that operate heavily dependent on the state of the tests in their context
- The Metrics Window with its code coverage percentages
- The various data exports throughout the software, covering everything from CI integration to simple coverage XML reports
I don't doubt I could sit down with everyone requesting this feature and they could each help me come up with a UX solution for each of the above areas of the software that sensibly considered their usage of an inconclusive test state, but the answers would all be different because everyone uses this state differently.
Even without considering the tremendous effort needed to implement support for such a massive cross-cutting concern right throughout the NCrunch codebase, and the effort needed to maintain so much extra complexity, I genuinely don't feel there is a 'right' way to implement this that wouldn't leave the majority of users very confused.
Joseph Musser commented
Inconclusive tests are perfect when you have integration tests that may or may not run. I am rather surprised I can't use NCrunch for this.
Toby Carter commented
Same here. I'll often add roughly named tests with an inconclusive assert as i think of scenarios to support, but then whilst working on implementations it is hard to tell what is a failing test, and what is a placeholder.