Test Execution not starting after new bucket of tests is given to a Test Agent - by G. Angelchov

Status : 

  Fixed<br /><br />
		This item has been fixed in the current or upcoming version of this product.<br /><br />
		A more detailed explanation for the resolution of this particular item may have been provided in the comments section.


1
0
Sign in
to vote
ID 785539 Comments
Status Closed Workarounds
Type Bug Repros 1
Opened 4/25/2013 12:19:41 AM
Access Restriction Public

Description

Hi,

We use Test Controller 2010/Test Agents 2010 infrastructure to execute automated tests. However, randomly a test agent "hangs" upon receiving a new bucket of tests. 

I noticed that when that happens, the QTAgent32.exe process goes up to 100% CPU Utilization, and the test progress in the Agent Status window stays at "Test Progress: 0/10 Completed". 
Although the first test from the bucket is listed in the "Currently running test: " field, this test never starts and the execution hangs. 

As a result the test run is never completing - although all other test agents finished execution of the remaining tests, the test agent in question stays in "Running Tests" forever. 

If I manually open the task manager and kill the QTAgent32.exe process, the test execution resumes normal operation and the rest of the tests are executed correctly.
Sign in to post a comment.
Posted by Venkata K Sadineni on 6/2/2014 at 12:26 PM
Hello,
Thank you for your response. I have tried with Visual Studio Ultimate 2013, Test Agents - 2013, Test Controller -2013. In the Remote.testsettings file, I have specified the BucketSize as 100.
<Buckets size="100"/>

I have specified the TestController name in the remote.testsettings file.
<RemoteController name="MyTestControllerServer" />

Test Env Setup: 2 test Agents each running Test Agent - 2013; They are registered to a Test controller - 2013

My test run setup is like this:
Suite1 - 1600 tests; Suite2 to Suite4 has - 350 tests each

Run below commands in four 4 different command line windows:
mstest /testcontainer:Suite1.dll /testsettings:remote.testsettings
mstest /testcontainer:Suite2.dll /testsettings:remote.testsettings
mstest /testcontainer:Suite3.dll /testsettings:remote.testsettings
mstest /testcontainer:Suite4.dll /testsettings:remote.testsettings

I started the Suite1.dll first followed by the rest.
After finishing the first Bucket(100) of tests on each agent, then agents do NOT execute any tests.

In the Test Controller logs, I see the below lines:
V, 7312, 10, 2014/06/02, 11:33:53.469, MytestControllerSerever\QTController.exe, ControllerObject: Adding agents to run 11025e07-1dda-4b5a-8aae-4981bb0acc91
V, 7312, 10, 2014/06/02, 11:33:53.469, MytestControllerSerever\QTController.exe, ControllerObject.AddAgentsToRun: the run 11025e07-1dda-4b5a-8aae-4981bb0acc91 has not started, trying reserve agents for it...
V, 7312, 10, 2014/06/02, 11:33:53.469, MytestControllerSerever\QTController.exe, AgentManager: ReserveAgents is returning 0 agents
V, 7312, 10, 2014/06/02, 11:33:53.469, MytestControllerSerever\QTController.exe, ControllerObject: No agents available for run 11025e07-1dda-4b5a-8aae-4981bb0acc91.

V, 7312, 10, 2014/06/02, 11:33:53.469, MytestControllerSerever\QTController.exe, ControllerObject: Adding agents to run b8f718e9-77c7-4e15-a4e3-9a7bf5b8ff7f
V, 7312, 10, 2014/06/02, 11:33:53.469, MytestControllerSerever\QTController.exe, ControllerObject.AddAgentsToRun: the run b8f718e9-77c7-4e15-a4e3-9a7bf5b8ff7f has not started, trying reserve agents for it...
V, 7312, 10, 2014/06/02, 11:33:53.469, MytestControllerSerever\QTController.exe, AgentManager: ReserveAgents is returning 0 agents
V, 7312, 10, 2014/06/02, 11:33:53.469, MytestControllerSerever\QTController.exe, ControllerObject: No agents available for run b8f718e9-77c7-4e15-a4e3-9a7bf5b8ff7f.

V, 7312, 10, 2014/06/02, 11:33:53.688, MytestControllerSerever\QTController.exe, ControllerObject: RunQueueThread waiting for agents...
V, 7312, 10, 2014/06/02, 11:33:53.688, MytestControllerSerever\QTController.exe, ControllerObject: An agent is available.
V, 7312, 10, 2014/06/02, 11:33:53.688, MytestControllerSerever\QTController.exe, ControllerObject: Adding agents to run d2b6ea2d-7739-4a0b-b0c4-2d111b7b1dc5
V, 7312, 10, 2014/06/02, 11:33:53.688, MytestControllerSerever\QTController.exe, ControllerObject.AddAgentsToRun: the run d2b6ea2d-7739-4a0b-b0c4-2d111b7b1dc5 has tests and it has already started, let's see if there are some free execution agents...

V, 7312, 10, 2014/06/02, 11:33:53.797, MytestControllerSerever\QTController.exe, ControllerObject: Adding agents to run f3b0b9b8-9316-4528-9cf6-e3b7fab3baaf
V, 7312, 10, 2014/06/02, 11:33:53.797, MytestControllerSerever\QTController.exe, ControllerObject.AddAgentsToRun: the run f3b0b9b8-9316-4528-9cf6-e3b7fab3baaf has not started, trying reserve agents for it...
V, 7312, 10, 2014/06/02, 11:33:53.797, MytestControllerSerever\QTController.exe, AgentManager: ReserveAgents is returning 0 agents
V, 7312, 10, 2014/06/02, 11:33:53.797, MytestControllerSerever\QTController.exe, ControllerObject: No agents available for run f3b0b9b8-9316-4528-9cf6-e3b7fab3baaf.
Posted by Microsoft on 4/29/2014 at 12:30 PM
Thank you for reporting this issue. This issue has been fixed in Visual Studio 2013. You can install a trial version of Visual Studio 2013 with the fix from: http://go.microsoft.com/?linkid=9832436
Posted by Venkata K Sadineni on 8/25/2013 at 12:38 AM
Hello Anudeep,
On VS11 (2012 Test Controller\Agent versions), I have tried the fix(work around) suggested by you. It did NOT help. I still see that the agent hangs after the first bucket is complete.

Posted by Venkata K Sadineni on 8/25/2013 at 12:19 AM
Hello,
I am also facing the same issue on VSTS2012 Test Controller\Agent setup. After the first bucket is complete, the test agent hangs executing the next set of tests. It says: "0/30 completed". Where 30 is the 2nd set of tests given to this agent.

Is there any fix for VS11?? Please help me to unblock this issue.
Posted by Microsoft on 7/9/2013 at 3:49 AM
Thanks for reporting the issue. We have published a patch(http://support.microsoft.com/kb/2801364) for Dev10 SP1 which should address your problem. Install patch 2801364 on client, Test Controller and Test Agent machine.

You should increase the timeout setting for Agent timeout:

In QTController.exe.config: <drive letter:>\Program Files (x86)\Microsoft Visual Studio <Visual Studio Version>\Common7\IDE\.
    <add key="AgentConnectionTimeoutInSeconds" value="120"/>
    <add key="AgentSyncTimeoutInSeconds" value="300"/>
Increase the value to triple.

In QTAgentService.exe.config <drive letter:>\Program Files (x86)\Microsoft Visual Studio <Visual Studio Version>\Common7\IDE

                    Set ControllerConnectionPeriodInSeconds to 90 (making this 3 times too).


This should resolve the issue. In case further help is required please feel free to contact us.

Thank you
Anudeep Sharma
Senior Dev Engineer, Visual Studio.
Posted by G. Angelchov on 5/9/2013 at 11:35 PM
Okay, the issue just manifested itself again. I'm attaching Controller/Agent logs as well as the event viewer logs for both.
You can find them in the attached .zip file - 05.10.2013.zip

The issue happened on May 9th around 19:34 - there are a few errors in the logs around that time but they don't say much.

Any help is highly appreciated.

Thanks,
-Georgi
Posted by G. Angelchov on 5/6/2013 at 11:29 PM
Hi,

The controller and agents are Virtual Machines that are reverted to a clean state before each test execution, so any errors in the application event logs are lost after reverting. That's the reason I cannot share the event logs at the moment. I'll do so the next time this issue happens, but this might take some time.


Regards,
-Georgi
Posted by Microsoft on 5/2/2013 at 9:11 PM
Hi Angelchov,

Can you please share the errors in the application event logs of your test controller as well as the test agent machine?

Regards
Visual Studio Product Team
Posted by Microsoft on 4/25/2013 at 3:00 AM
Thanks for your feedback.

We are rerouting this issue to the appropriate group within the Visual Studio Product Team for triage and resolution. These specialized experts will follow-up with your issue.
Posted by Microsoft on 4/25/2013 at 12:51 AM
Thank you for your feedback, we are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)