I'm seeing some very strange behavior with the GC under C#. I'm seeing it
in two different applications that have nothing in common except that they
both use and release a lot of memory in BYTE or CHAR arrays.

The first one is easier to describe, but the behavior is similar in both.

WinXP on a machine with 512Meg and NO swap file (for performance).

I'm using a winForms app as a testing harness against some networking code
that stores results in a MemoryStream. The results are processed, then all
objects are released. Watching system memory in Task Manager shows that
memory is getting eaten up, but never freed. Because there's no swap file,
Windows can't increase the memory, so the application will eventually crash
with an out-of-memory error. Interestingly enough, it will actually do this
even though it has plenty of memory waiting to be collected.

The MemoryStream class increases capacity as needed by allocating a new
buffer of double the existing size and copying the data to the new buffer.
I'm starting it at one 1MB, so it goes 1,2,3,4,8,16,.... When it gets to
the point where it needs to allocate a 256 meg block, it will fail with a
memory error because there is only about 200MB of system memory left. The
annoyance is that it actually has enough memory, if it would just collect
the old buffers. You see, at this point, the MemoryStream is using a 128MB
buffer, but has 127MB in discarded buffers (1+2+4+8+16+32+64). If I throw
in calls to GC.Collect(), it works fine.

I had first thought that the GC just wasn't collecting because I was using
all of the CPU, but it just never seems to collect. After a smaller run, I
left the testing harness up for 30 minutes after everything has been freed,
and it just doesn't collect, however, if I put a call to
GC.GetTotalMemory(true) on a button, it instantly frees all of the memory.
GC.GetTotalMemory() reports 83Meg used then GC.GetTotalMemory(true) shows
461K.

I have spread calls to GC.Collect through the code so solve the problem, but
it seems foolish to me. Any thoughts?

By the way, this happens inside and outside of VS as well as in code
compiled for Release. And, of course, I already have a call to
IDisposable.Dispose() on the MemoryStream.

#1: having no swap file isn't the best idea in the world, as the swap file
is also used for a backing store for various operations. Windows will not
swap unless it thinks it needs it. Often, it will pre-emptively swap and
also leave the pages in memory, so it can instantly reclaim the memory pages
for cache, improving performance quite a bit. In other words, having a swap
file is a good thing and not having one doesn't buy you much in performance.
I have 1GB of mem on my XP machine and a 1GB swap file and my computer is
snappy as can be.


#2: are you sure that you are releasing the reference to the memorystream
object after each run? If the instance of the class is still sitting around
holding a reference that could be a problem. Alternately, your app may
simply have a memory usage pattern that isn't well-recognized by the GC,
although the GC is definitely supposed to collect more often as system
memory runs out. So in this case, perhaps there is a bug.

-- russ

"John Smith" <Jon41 (AT) aol (DOT) com> wrote

"Russ Bishop" <nowhere> <"Russ Bishop" <nowhere>> wrote:
Unfortunately, it doesn't always get it right. If I have a large
program (such as an IDE) and I don't use it for a while (eg over lunch)
Windows usually decides to swap it out in favour of more file buffers -
and then the app grinds horribly when I come to use it again. I wish
you could selectively decide that a particular app should *never* be
swapped out unless it's *absolutely* necessary.

Certainly if GC.Collect() can collect the memory but it doesn't under
normal operation, it sounds like something odd is going on.

--
Jon Skeet - <skeet (AT) pobox (DOT) com>
http://www.pobox.com/~skeet/
If replying to the group, please do not mail me too

John Smith <Jon41 (AT) aol (DOT) com> wrote:
I might try that some time - I always use a laptop...

I've had to up it to 512Mb to see it garbage collect, but then it does.
Not as well as if you call GC.Collect() yourself, but I'll explain why
I think that's happening in a second.

Here are a few sections of my trace without manual collection:

Blocks: 43,200 Bytes: 168M Capacity: 256M Memory: 480M
Blocks: 43,300 Bytes: 169M Capacity: 256M Memory: 384M
Blocks: 43,400 Bytes: 169M Capacity: 256M Memory: 256M

That shows that some GC is happening.

Now, I *am* seeing some behaviour I don't understand, but I think I can
explain some of why it's so much better with manual collection
immediately *after* allocation. During the write itself, the stream
needs to do the following:

1) Realise its buffer is full
2) Create a new buffer
3) Copy the old buffer into the new

Step 2 is the natural place to garbage collect, but the old buffer is
still in use. After that, there's no reason to garbage collect (as far
as the GC thinks) because nothing's needing more memory. When you
garbage collect manually, it's *after* step 3, so the old buffer is no
longer needed.

This doesn't explain *everything* though - it doesn't explain why I
have seen a total memory of 896M (128+256+512) rather than always 768M
(256+512) which I'd expect. However, it does explain why if I allocate
a megabyte (enough so it can't be in gen0) manually just before your 5
second sleep, just the act of allocating that memory is enough to prod
the GC:

Blocks: 65,600 Bytes: 256M Capacity: 512M Memory: 896M
Blocks: 65,700 Bytes: 256M Capacity: 512M Memory: 896M
Blocks: 65,792 Bytes: 257M Capacity: 0M Memory: 896M
Allocating some memory to prod the GC
Blocks: 65,792 Bytes: 257M Capacity: 0M Memory: 1M

Now, the GC can also respond to pressure from the rest of the OS for
memory. Running two copies of gc_check in parallel (which is fairly
hideous for the system, admittedly I see the memory use go down much
earlier.

So it sounds like the GC could possibly be more aggressive at
periodically checking if it knows that it's got a large proportion of
system memory allocated, but it *is* responding to pressure/allocations
in the right way.

--
Jon Skeet - <skeet (AT) pobox (DOT) com>
http://www.pobox.com/~skeet/
If replying to the group, please do not mail me too