I have tried to use sos.dll together with the WinDbg debug engine to perform
some automated retrieval of runtime data. Unfortunately, there seems to be a
major memory leak in the version of sos.dll that is shipped with .NET
Framework 2.0. Each time an sos command is executed, e.g !DumpDomain,
!DumpModule or !DumpMT, the memory consumption of WinDbg increases by about 5
MB. This memory is never released, and since I want to execute a large amount
of commands (> 500) the process quickly runs out of memory.

This error only occurs in the v2.0 sos.dll. The sos.dll provided with the
Debugging tools (for v1.x) seems to be working correctly.

Is there any possibility that this problem will be fixed soon? E.g will an
updated sos.dll be included with the debugging tools download?

Hi ASuurkuusk,

Thanks for your post.

I have tried to use "Microsoft .NET Framework SDK v2.0" command prompt to
lauch the windbg and load SOS.dll. However, in windbg, !DumpModule or
!DumpMT does not cause any significant increase in "Task Manager" "VM size"
value.

Have you tried other VS2005 machine's SOS.dll? If this problem is
reproducable on other machine, I suggest you provide the detailed steps to
help us reproduce the problem. Thanks

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Hi,

I have tried this on two machines, both with Windows XP SP2 as well as under
a VMWare session. One machine was running a Swedish OS and VS 2005
Professional and one was running an English OS and VS 2005 Team Developer. I
have also tried with all downloadable versions of WinDbg (v6.6.3.5, v6.5.3.8,
v6.4.7.2, v6.3.17.0), except v6.2.13.1. The behaviour is the same.

To reproduce this behaviour start WinDbg and then start or attach (invasive
or non-invasive) a .NET executable (e.g an empty Windows Forms application).
Then load sos.dll and issue some commands. After each command the memory
usage of WinDbg increases by about 5MB.

To show you this behaviour I recorded a movie from a VMWare session. This
session runs a freshly installed Windows XP SP2 OS, with .NET Framework 2.0
and Debugging Tools for Windows v6.6.3.5 installed. The .NET application used
is just an empty Windows Forms application created using VS2005. The Task
manager in the movie shows the memory usage of WinDbg, as well as the total
Commit charge.

The movie can be downloaded from: http://www.scitech.se/temp/SosLeak2.avi

Best regards,

Andreas Suurkuusk
SciTech Software AB

""Jeffrey Tan[MSFT]"" wrote:

Jeffrey,

Did you try !DumpDomain?
This effectively leaks 5MB and more depending on the number of domains in
the process.

Willy.

""Jeffrey Tan[MSFT]"" <jetan (AT) online (DOT) microsoft.com> wrote

Hi Andreas and Willy,

Thanks for your feedback.

Yes, I can reproduce out this behavior, I will feedback this issue to the
product team, I will reply any progress here. Thanks

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Hi Andreas ,

Sorry for letting you wait.

Based on the discuss with the product team, they confirmed that the 2.0 SOS
just makes calls into the framework to get the data and the framework gets
loaded in windbg process.

Also, some of the commands cache data so that subsequent lookups don't take
as long. So this is not identified as a leak.

Hope this helps!

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Hi,

I did perform some debugging into sos.dll before reporting this issue, and I
am aware that the runtime dll and mscordacwks.dll get loaded into the windbg
process and that some data gets cached. However, this is not the problem. The
problem is that the cache does not get reused between sos commands. If you
looked at the movie I provided (http://www.scitech.se/temp/sosleak2.avi), you
would have seen that I issued a DumpMT command for the same method table
(7b473dfc) several times. Each time the memory usage of windbg.exe increased
with about 6 MB. If the data about the provided method table was cached, then
the memory usage should certainly not increase.

In order to provide you with some more information about this problem, I
performed some additional debugging of sos.dll and dbgeng.dll. I managed to
pinpoint the problem to the LoadClrDebugDll function in sos.dll. The
LoadClrDebugDll gets called each time an sos command is performed. It makes a
call to dbeng.dll!ExtIoctl (using ExtensionsApis.lpIoctlRoutine) with
IoctlType set to 0x26, which will create a new "CLRData" instance using
CLRDataCreateInstance. This new instance is then assigned to g_clrData,
overwriting the previous instance of "CLRData", which I believe causes the
memory leak. I performed a test where I modified the LoadClrDebugDll function
so that it only runs once (on subsequent calls it returns immediately). This
avoids creating and leaking "CLRData" instances and it seems like sos.dll
works correctly, without leaking any memory.

At the end of this post I have provided my (partially and manually)
decompiled code for LoadClrDebugDll and ExtIoctl. I have marked the sections
where I believe there is a problem with "PROBLEM:".

I don't know if it will cause any side effects, but adding "if( g_clrData !=
NULL ) return;" to LoadClrDebugDll, will probably avoid the memory leak.

Another thing that I have noticed, is that this memory leak doesn't occur
when sos.dll is loaded into VS 2005. I have not investigated this further,
but since VS doesn't use dbgeng.dll, I assume that ExtIoctl (0x26) in the VS
implementation will return the same "CLRData" instance each time it's
requested.


Best regards,

Andreas Suurkuusk
SciTech Software AB

struct CLRDebugData
{
IID* iid;
void** iface;
}

IID IID_IClrData = 5c552ab6_fc09_4cb3_8e36_22fa03c798b7

SOS.dll!LoadClrDebugDll()
{
CLRDebugData data;
data.clsId = &IID_IClrData;
if( ExtensionsApis.lpIoctlRoutine( 0x26, &data, sizeof( CLRDebugData ) )
!= 0 )
{
// PROBLEM: Will overwrite previous g_ClrTarget (memory leak ~ 5MB!)
g_clrData = data.iface;
}
}

ULONG dbgeng.dll!ExtIoctl(USHORT IoctlType, PVOID lpvData, ULONG cbSize)
{
switch( IoctlType )
{
case 0x26:
if( g_Process != NULL && cbSize == sizeof( CLRDebugData ) )
{
CLRDebugData* data = (ClrDebugData*)lpvData;
g_Process->LoadClrDebugDllForExt();
if( g_Process->pCLRDataCreateInstance != NULL )
{
// PROBLEM: Will create a new IClrDataTargetInstance each time
if( g_Process->pCLRDataCreateInstance( *data->iid,
g_Process->target, data->iface ) == S_OK )
return (ULONG)TRUE;
}
}
break;
case ...:
break;
}
...
}

""Jeffrey Tan[MSFT]"" wrote:

Andreas, Great analysis, which I'm happy to confirm.

Willy.


"Andreas Suurkuusk" <ASuurkuusk (AT) newsgroups (DOT) nospam> wrote

Hi Andreas,

Very cool!

I have forward your debugging information to the sos.dll owner, I will
feedback any progress here.

Personally, I am curious about some of your debugging details.

1. Do you get the source code of the functions in dbgeng.dll by
disasssembling with symbols in windbg?
2. If you can do reverse engineering to the function, how do you "adding
"if( g_clrData != NULL ) return;" to LoadClrDebugDll"? Do you modify the
executing process in windbg in debugging-time?

Thanks

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Hi,

1. Actually I prefer the VS 2005 debugger and only use windbg when I have
to. Anyway, the source code was manually decompiled using disassembly,
together with symbol files for dbgeng.dll and sos.dll. The symbol files
provided me with some function and variables names. The IID_IClrData and
CLRDebugData names I made up my self.

2.I have written my own wrapper around dbgeng.dll, and the modification I
mentioned was performed by replacing the first instructions of
LoadClrDebugDll with "xor eax, eax" and "ret". The modification was performed
after having called an sos command once, so I didn't actually have to add
"if( g_clrData != NULL )". In my first attempts I replaced the instructions
manually within the debugger, but then I added the code below to my dbgeng
wrapper:

pDebugControl->Execute( 0, cmd, 0 );
if( cmd[0] == '!' && firstCommand )
{
byte* pLoadClrDebugDll = (byte*)0x64280220; // Hard coded start address of
LoadClrDebugDll

DWORD oldProtect;
VirtualProtect( pLoadClrDebugDll, 3, PAGE_EXECUTE_READWRITE, &oldProtect );

pLoadClrDebugDll[0] = 0x33; // xor eax, eax
pLoadClrDebugDll[1] = 0xc0; //
pLoadClrDebugDll[2] = 0xc3; // ret

VirtualProtect( pLoadClrDebugDll, 3, oldProtect, NULL );
FlushInstructionCache( GetCurrentProcess(), pLoadClrDebugDll, 3 );
firstCommand = false;
}

Best regards,

Andreas Suurkuusk
SciTech Software AB

""Jeffrey Tan[MSFT]"" wrote: