Visual Studio and .NET Framework Home
Hang loading rasapi32.pdb when using symbol server
3/12/2009 11:38:24 AM
User(s) can reproduce this bug
When running one of our components in the visual studio debugger, the symbol server hangs downloading rasapi32.pdb. The problem seems to occur when our component is loading the WebBrowser control. We started seeing this hang after upgrading to Windows XP SP3, though I think that's a red herring and the SP simply upgraded some DLLs that now need new symbols. There also needs to be some content in the control (about:blank doesn't trigger the problem) to get a hang, but again I think that's just because one has to get IE to load some DLLs I don't already have symbols cached for.
I believe this is the same problem that Steven Wilssen replied to at http://social.msdn.microsoft.com/Forums/en-US/refsourceserver/thread/26a4b712-6c82-4627-90f8-7188c7ebab10/, where he indicated he was seeking a repro case for this. However, my attempt to contact him bounced with "Recipient address rejected: Access denied"
My theory is that this is a deadlock in trying to fetch http resources:
1) the embedded is loading a number of helper dlls as it downloads the page content;
2) the symbol server fails to find cached symbols for at least one of these (as in the linked forum post, the one that hangs for me is rasapi32.dll)
3) it then tries to connect to http://msdl.microsoft.com/download/symbols, but ends up waiting for something internal to wininet/urlmon that is already held by the web browser control's incomplete download, which is not making progress because the debugger has stopped the process to load symbols.
IE also hangs trying to browse to any site while the machine is in this state, but Firefox still works, so I think it's some sort of system-wide lock in wininet HTTP handling. Lacking much knowledge of wininet internals, that's the best I can do for a theory.
Visual Studio 2005 SP1
Windows XP Professional
Operating System Language
Steps to Reproduce
Enable the microsoft symbol server (set the environment variable _NT_SYMBOL_PATH=C:\Data\symbols*http://msdl.microsoft.com/download/symbols, or use the visual studio GUI to set up http://msdl.microsoft.com/download/symbols)
Clear the symbol server's cache of pdb files (or at least make sure rasapi32.pdb is missing from it).
Build and run the project in PDBLoadHang.zip in the debugger.
My system is Visual Studio 2005 SP1 on Windows XP SP3 with IE7.0.5730.13CO
The crashdump in devenv.zip should have all the specific file versions listed.
the debugger loads numerous symbol files and begins loading symbols for rasapi32.dll, but does not make any more progress. Hitting break/stop after this point causes the visual studio IDE to hang.
devenv.zip has a crashdump of the deadlocked visual studio process. This was rather tricky to get, because any attempt to attach to devenv *also* deadlocked while loading symbols - the problematic lock appears to be system-wide. I turned off the symbol server in the second visual studio instance before attaching to the first one to get this, saved the crash dump, killed everything so wininet would come back to life, then opened the crash dump with the symbol server on so I'd get a good stack trace.
I think the stuck thread is 2592, which has the call stack:
ntdll.dll!_NtWaitForSingleObject@12() + 0xc bytes
kernel32.dll!_WaitForSingleObjectEx@12() + 0x8b bytes
kernel32.dll!_WaitForSingleObject@8() + 0x12 bytes
wininet.dll!FixProxySettingsForCurrentConnection() + 0x25 bytes
wininet.dll!CFsm_HttpSendRequest::RunSM() + 0x3c bytes
wininet.dll!CFsm::Run() + 0x30 bytes
wininet.dll!DoFsm() + 0x25 bytes
wininet.dll!HttpWrapSendRequest() + 0x312bc bytes
wininet.dll!_HttpSendRequestW@20() + 0x5d bytes
symsrv.dll!StoreWinInet::request() + 0x2a bytes
symsrv.dll!StoreWinInet::fileinfo() + 0x18 bytes
symsrv.dll!StoreWinInet::open() + 0x7c bytes
symsrv.dll!StoreWinInet::find() + 0xb8 bytes
symsrv.dll!cascade() + 0x87 bytes
symsrv.dll!_SymbolServerByIndexW@16() + 0x127 bytes
symsrv.dll!SymbolServerW() + 0x77 bytes
mspdb80.dll!LOCATOR::SYMSRV::SymbolServer() + 0x16f bytes
mspdb80.dll!LOCATOR::FLocatePdbSymsrv() + 0x71 bytes
mspdb80.dll!LOCATOR::FLocatePdbPathHelper() + 0x175 bytes
mspdb80.dll!LOCATOR::FLocatePdbPath() + 0xf4 bytes
mspdb80.dll!LOCATOR::FLocatePdb() + 0x133 bytes
mspdb80.dll!PDBCommon::OpenValidate5() + 0xa9 bytes
msenc80.dll!enc::EncImpl::~EncImpl() + 0x32af bytes
NatDbgDE.dll!OLPDBOpen() + 0x7e bytes
NatDbgDE.dll!OLStart() + 0xfd bytes
NatDbgDE.dll!LoadOmfForReal() + 0x22 bytes
NatDbgDE.dll!LoadSymbols() + 0x40 bytes
NatDbgDE.dll!OLLoadOmf() + 0x4c bytes
NatDbgDE.dll!SHLoadDll() + 0xad bytes
NatDbgDE.dll!CSymbolHandlerX::SHLoadDll() + 0x58 bytes
NatDbgDE.dll!CModule::Load() + 0xd6 bytes
NatDbgDE.dll!CNativeProcess::NotifyModLoad() + 0xce bytes
NatDbgDE.dll!EngineCallback() + 0x8e bytes
NatDbgDE.dll!_EMCallBackDB@32() + 0x2c bytes
NatDbgDE.dll!LoadFixups() + 0x20c bytes
NatDbgDE.dll!DebugPacket() + 0xa4 bytes
NatDbgDE.dll!EMFunc() + 0xba62 bytes
NatDbgDE.dll!_TLCallBack@16() + 0x1c bytes
NatDbgDE.dll!TLClientLib::Local_TLFunc() + 0xb1d9 bytes
NatDbgDE.dll!DMSendPacket() + 0x82 bytes
NatDbgDE.dll!NotifyEM() + 0xda bytes
NatDbgDE.dll!ProcessLoadDLLEvent() + 0x33 bytes
NatDbgDE.dll!ProcessDebugEvent() - 0x362 bytes
NatDbgDE.dll!DmPollLoop() + 0xca45 bytes
msvcr80.dll!__endthreadex() + 0x3b bytes
msvcr80.dll!__endthreadex() + 0xc7 bytes
kernel32.dll!_BaseThreadStart@8() + 0x37 bytes
Debugger downloads the symbol files and can be used to debug the process.
TAP Code (if applicable)
You can indicate your satisfaction with how Microsoft handled this issue by completing this quick
3 question survey
to post a comment.
Please enter a comment.
on 6/14/2012 at 1:26 AM
No fix yet, Win7, VS2010...
on 7/17/2010 at 7:25 PM
Experienced this issue with a CLI app., symchk'ed entirety of \SysWow64 and \System32 to workaround.
on 3/17/2009 at 4:03 PM
Thank you for taking the time to report this, we believe that you hit the nail on the head:
1) Application being debugged contains web browser control
2) Web browser controls loads rasapi under a system wide lock
3) DllLoad is received by debugger (debuggee is in break mode)
4) Debugger needs to make an internet request to symbol server
5) Wininet needs to take the lock the debuggee already holds in order to talk to the symbol server
Unfortunately there is really nothing the debugger can do about this except block the download for the one binary is loaded under the lock. We believe the better solution is for Windows not load the binary while holding this lock. Since the windows team uses a different bug tracking database than Visual Studio, I am going to resolve this bug as “External” and file a bug against the Windows team.
Visual Studio Debugger
on 3/13/2009 at 2:05 AM
Thanks for your feedback.
We are escalating this issue to the appropriate group within the Visual Studio Product Team for triage and resolution. These specialized experts will follow-up with your issue.
Visual Studio Product Team
to post a workaround.
Please enter a workaround.
on 12/30/2011 at 3:02 AM
In VS2010 you can get the Windows Help viewer to open with the symbol server active if you add the following to the excluded module list:
on 12/7/2009 at 12:22 PM
You may need to do the above step with rasman.dll and possibly others, as well.
on 12/7/2009 at 12:18 PM
You can manually download the file via symchk from Debugging Tools for Windows, for example:
C:\Program Files\Debugging Tools for Windows (x64)\symchk.exe c:\windows\SysWOW64\rsapi32.dll
Modify as necessary for your OS.
on 3/12/2009 at 11:39 AM
If, instead of running the process in the debugger from the start, I allow it to launch and show the webbrowser control, then attach the debugger, the symbols download normally, and subsequent runs starting in the debugger work too (because the symbols are now cached).
© 2014 Microsoft