Home Dashboard Directory Help
Search

Test Explorer Gives invalid feedback during runs. by Andrew Ritz


Status: 

Closed
 as By Design Help for as By Design


1
0
Sign in
to vote
Type: Bug
ID: 751311
Opened: 6/28/2012 7:36:29 AM
Access Restriction: Public
0
Workaround(s)
view
1
User(s) can reproduce this bug

Description

When tests are running, the test explorer gives invalid, and sometimes misleading feedback. In the screenshot I have supplied we can see that the test run is in progress, but it also suggests that all 51 tests (in this case) have passed.

It would seem to me that the tests that haven't been run yet should be listed as "Not Run Tests", etc. For cases where there are hundereds of tests, it is basically impossible to figure out what the test runner is doing at a glance. This confusion is compounded even more if there are "Failed Tests" or "Skipped Tests" that haven't actually run yet.
Details
Sign in to post a comment.
Posted by Andrew Ritz on 7/9/2012 at 5:47 AM
While it is nice that the status of individual tests are updated as they are run, like I said before, the Test Explorer doesn't show "complete" information until ALL of the tests have been run. Any of the tests that have not been run are essentially in an indeterminate state, but the explorer lists them as "passed/failed/whatever". Again, this makes the information presented invalid at a quick glance.

Meanwhile, out of everyone in my organization who does testing on a regular basis, we all agree that nobody is interested in "tracking individual tests" during a run. If we are, then we just run that test by itself. Also, there is a search bar at the top of the existing test explorer which would resolve the supposed "lost test" issue. (unless it uses bing :) )

Anyway, since you guys are married to all of these poor design choices, I'll just use some other test runner / framework.
Posted by Microsoft on 7/3/2012 at 12:15 PM
Hi Andrew,

The Visual Studio Test Explorer updates the status of each test as it is executed, not only at the end of a run. This is most obvious if you create a solution with multiple long running test.

Until the test is run it displays the results from the last execution. I still feel this is the ideal behavior. If we were to move all tests to the Not Run group. And then move them to the Run with status as execution completes it would be very difficult to track individual tests, as they would be moving groups constantly (every test would move at least twice during every run). For projects with a small number of tests this may be resonable. But for projects with 100s or 1000s of tests, moving the test between groups will cause it to become lost. The trade off is that the test displays the previous run status, while waiting to be executed. For the primary case of tests, that they return the same results as the previous run it never moves and is easier to locate.

Thank you for your feedback,
Joshua Weber
Posted by Andrew Ritz on 7/2/2012 at 5:14 AM
"some potential for lag"? Are you kidding me? The only time the test runner is displaying complete information (by design) is when it done running all of the tests? I think that your "motion" excuse is a pretty poor one, as anyone would rather have a display that contains useful information, as opposed to one that does not. So now we are all stuck with a test runner that produces unreliable and nebulous results. Thanks!
Posted by Microsoft on 6/29/2012 at 8:58 AM
Andrew,

Thank you very much for this feedback. The Test Explorer updates the status, results, and category of each test individually as it is run. Because it takes some amount of time to execute all tests this does create some potential for lag between the start of execution and the updating of all tests. I understand your feedback of wanting a more clear display. We feel that we must balance the amount of movement of tests within the Test Explorer. If the Test Explorer window was to move all tests to a not run state, and then move them as results came in the window would be going through significant motion. Every test would be moved twice during each run. In the most common case only a few of many test change state each run. We feel it is better to introduce the slightly delay and mismatch, then to create the high motion and potentially distracting view in the Test Explorer window.

Thanks for your great feedback,
Joshua Weber
Posted by Andrew Ritz on 6/29/2012 at 6:33 AM
Again, it would be nice if you would actually post the link to the supposed "duplicate" bug!
Posted by Microsoft on 6/29/2012 at 1:32 AM
Thanks for your feedback.

We have marked this bug as a duplicate of another issue we are already tracking. Please see the issue linked to this bug for updates.

Thank you,

Microsoft Visual Studio Connect Support Team.
Posted by Microsoft on 6/28/2012 at 7:48 AM
Thank you for your feedback, we are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)
Sign in to post a workaround.
File Name Submitted By Submitted On File Size  
invalid-results.png (restricted) 6/28/2012 -