Set a baseline in a project with many failing tests
Here's a new idea: I'd like to be able to set a "failure baseline" in a project with a lot of failing tests.
Granted, such a project shouldn't exist in the first place! But I'm sure many of us have had the experience of getting to a new workplace where nobody's run the unit tests for a while, and suddenly the new guy runs them and there are hundreds failing.
Of course, the ideal thing would be to stop everything and fix them all. But try explaining to management why we can't fix bugs or write new features for a few weeks.
Thus, my proposal:
I fire up NCrunch, do a complete test run, and find out that 637 tests are failing. Now I can tell NCrunch to save that as a baseline. NCrunch will store the details on each of those 637 failing tests. Then when I modify code, and NCrunch reruns tests, it'll give me baseline-compared results (perhaps in a new window, separate from the normal test result window) telling me about (1) any new failing tests, and (2) any failing tests in that baseline that now pass.
Currently, if I write code that makes a failing test to pass but also makes a formerly passing test fail, I'll just see the same number, 637. With this new feature, I'd see a clear indicator that compared to the baseline, I have 1 more failing test and 1 more passing test, and I can see exactly what those tests are.
One question is how this would work with branches. I assume NCrunch isn't really branch-aware, but simply focuses on the current state of the code. The baseline should probably be a file saved in the project, such as an XML or JSON file. Then on a different branch I can keep the same file or overwrite it with a different baseline file if I so choose.
