Search

Memory leaks and performance drop on TFS 2010 Build Service under heavy load by Roman.Voropaev

Closed
as Fixed Help for as Fixed

39
0
Sign in
 to vote
Type: Bug
ID: 597476
Opened: 9/13/2010 7:57:47 AM
Access Restriction: Public
1
Workaround(s)
27
User(s) can reproduce this bug
Hello,

We've experienced an annoying problem with our TFS 2010 Build Service. We have a dedicated build machine (Xeon X5550 and 3GB RAM) for Continuous Integration builds. We have a mid-size code-base with mixture of C# (~80%) and C++ (~20%) projects. The projects and solutions are VS 2008 format at the moment. Due the geographical distribution of our developers we have check-ins (so triggered CI builds) roughly consistent 24 hours per day. The problem we have is that after a day or two memory usage on that build machine goes up (while performance goes down) until in 2-3 days builds stuck. The memory consumption of TFSBuildSericeHost.exe Process at that moment is usually 2.3-2.4 GB. Restarting the TFSBuildServiceHost Service releases the memory and solves the problem for 1-2 days again.

Is it a known problem? Are there any solutions/workarounds for that?

Kind regards,
Roman Voropaev
Details (expand)

Visual Studio/Silverlight/Tooling version

Visual Studio 2010

What category (if any) best represents this feedback?

Performance

Steps to reproduce

1) Have a dedicated machine that runs TFS Build Agent (Xeon X5550 and 3GB RAM in our case).
2) Have a mixed C# (80%) and C++ (20%) medium-size set of VS 2008 solutions to compile (~15 min for CI build in our case)
3) Have a Continuous Integration Build Definition (so triggered by check-ins). No gated check-ins.
4) Have check-ins roughly consistent for 24 hours per day (due to geographical distribution of developers).
5) After a day or two memory usage goes up (while performance goes down), until in 2-3 days build stuck. The memory consumption of TFSBuildSericeHost.exe Process at that time is usually 2.3-2.4 GB. Restarting the TFSBuildServiceHost Service releases the memory and solves the problem for 1-2 days again.

Product Language

English

Operating System

Windows Server 2008

Operating System Language

English

Actual results

Build agents works fine 1-2 days only. Then memory usage goes up and performance goes down until builds are stuck.

Expected results

Build agent works ok few weeks in a row without performance drop.
File Attachments
0 attachments
Sign in to post a comment.
Posted by Jason Intardonato on 2/27/2012 at 10:06 AM
I've seen this problem in every environment I've worked in using Team Build 2010 for the past 3 years. Currently have one build server running 10.0.30319.1 and another running 10.0.40219.1 and we see performance degradation on both servers, rather quickly RAM consumption increases and does not release even after builds have completed and there's no other activity on the server. Rebooting the server or restarting the process restores up to 2 GB of RAM.
Posted by jacobsmc on 11/28/2011 at 12:53 PM
We are still seeing this problem. Since SP1 we have seen #some# improvement, however we still require a restart of the build host process due to a degredation in build performance once the build process has consummed all the avaliable memory on our build server - this is now typically a weekly event.
Posted by jkbk171234 on 2/18/2011 at 8:13 AM
We are experiencing the same problem. We have a CI footprint amounting in <30 builds per day. The build service host process continually increases in memory untill it reaches over 2GB and then it stops responding (usually in 1-2 days). After restarting the service, it only takes a few hours for the memory consumption to increase significantlyand performance to decrease. This posting is currently marked as "Closed as Fixed", is this because there is a workaround or an actual solution? The workaround partially solves the issue, but for us the service would need to be restarted more rather frequently to improve the performance. Please post if there are any solutions or updates to this problem. Thanks!
Posted by Vladimir Gusarov on 12/17/2010 at 6:32 AM
It was clamed as fixed in SP1 beta.
Posted by blast461 on 11/23/2010 at 7:52 AM
Please provide information on this Fix. I am having these exact same problems and the workaround is not acceptable because of the build volumes our environment has.
Posted by Geoffrey Knoll on 11/18/2010 at 12:38 PM
Is this fix available as a KB somewhere that we can download it?
Posted by Anton Plotnikov on 11/12/2010 at 3:12 AM
I need the patch.
Posted by Cyril_Durand on 11/4/2010 at 3:50 PM
any updates ?
Posted by Mike Dang on 10/18/2010 at 9:26 AM
Same problem here. Patch please soon!
Posted by Don Sherwood on 10/12/2010 at 8:40 AM
Same problem here. Our build server has 12 GB RAM, and TFSBuildServiceHost happily consumes it all over the period of several days.
Posted by Vladimir Gusarov on 10/4/2010 at 1:05 PM
In my situation I can't even restart the service - it's doing something with 100% page faults. Giving more memory doesn't help.
Posted by Microsoft on 9/14/2010 at 12:28 AM
Thanks for your feedback.

We are rerouting this issue to the appropriate group within the Visual Studio Product Team for triage and resolution. These specialized experts will follow-up with your issue.

Posted by Microsoft on 9/13/2010 at 5:01 PM
Thank you for your feedback, we are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)
Sign in to post a workaround.
Posted by Svetlin Nakov on 9/16/2010 at 7:29 AM
You could create a small script to stop the TFSBuildServiceHost.exe process and start it again every night. Not really smart, but should work until MS release some fix.
© 2012 Microsoft Corporation. All rights reserved. Microsoft Connect Terms of Use | Trademarks | Privacy Statement