Allow specflow scenario outline examples to have their own 'dot marker' in the margin
currently if you have a scenario outline that has several examples there is no way to run a specific example as a test, I can only see the dots in the actual scenario outline and have to then run 'x' number of tests at once (where 'x' is the number of scenario examples)
I'd like a 'dot' by each line in the examples so I could run a specific scenario example on its own or see which example is actually failing
-
Sam Holder commented
I've validated that this can indeed be fixed from the specflow side (at least for MSTest tests) and will be submitting a pull request for specflow shortly.
I'm assuming that frameworks that allow [DataRow] annotation (which specflow uses to generate the Scenario Outlines in the compatible frameworks) would not be able to make use of this functionality as there is no difference in the lines of code that are actually executed in these cases (unless it can tell that it executed a method because a particular [datarow] annotation existed.
-
I admit that I'm having trouble visualising exactly what you mean, but it may well be possible to do this from the side of SpecFlow. The '#line' directives written by SpecFlow override the compiler's line numbering system, allowing it to 'pretend' that the lines are represented elsewhere. Assuming its possible to do what you're suggesting by adding more #line directives, then this may be solvable from the SpecFlow side.
-
Sam Holder commented
Thanks for the reply Remco. There seems to be a 1-to-1 mapping between the scenario examples and the variant number in the test name (and the TestProperty attribute on the test). ie the first line of the scenario examples is Test_Variant0, the second is Test_Variant1 etc, so I'd thought that the existing SpecFlow integration which allows the dots to appear in the Scenarios could be extended to place a dot by the examples which corresponds to the test that ran.
Does NCrunch use meta data output by specflow to allow integration with the feature files? Could SpecFlow be extended to output the required data to allow this?
-
Sadly this may very well be impossible to do with NCrunch.
NCrunch is reliant on debug metadata emitted by the compiler in order to map coverage data to physical lines of code. Where this debug metadata doesn't exist, it isn't possible for NCrunch to tell which 'line' of code was executed.
It's for this reason that we don't see markers on C# constructs that do not have a line numbered representation in the PDB file (for example, there are no markers sitting next to class or method declarations, despite this being quite heavily requested in the past).
The only way to get around this problem is by physically parsing source code (i.e. writing the first few steps of a compiler). The expense of such an approach makes it technically infeasible for a tool like NCrunch.