C# and the virtual memory "(3rd) party" OOM

In my new role I've been working on projects that are coded with C# and I was recently asked to look into an out of memory issue.  Being new to C# and not as attuned to the way in which it handles memory I turned to Google and Stackoverflow for some research.  I found out that C# in many ways is similar to Java in how it handles memory management, however, among the differences there is one big one - the unmanaged memory partition.  The memory isn't truly "unmanaged", it's just not directly managed by the running application and is typically stored in our virtual page space.  A scenario where this comes into play is the utilization of 3rd party COM objects within your program - it is here where our OOM issue begins to take shape.

Unmanaged memory

As we mentioned above this memory space isn't truly unmanaged in fact if you cleanup your references then the unmanaged memory will be allocated to the finalizer for cleanup; however, in practice (like with this issue) the garbage collector may not collect and cleanup the finalizer fast enough, thereby resulting in an OOM.  I found this strangely similar to a few issues I've observed in the JVM permgen space where code was being JIT compiled into the permgen space and not properly or timely released resulting in a permgen OOM issue.

Detecting the issue

If you have access to a profiler, such as Red Gate's Ants Profiler, then you can run a memory profile and do snapshot comparisons which would show you your unmanaged space growing.  You could also dive into what, at a high level, is in the unmanaged space and what objects are referencing that. 

If you don't have access to a profiler, then at a high level you can view the performance counters within your task manager for the process paying special attention to memory (working set), handles and virtual memory (paged pool).  As you conduct the steps to reproduce the issue you will probably see these increase without ever coming back to "normal" levels.  

If you observe the results indicated in either of these methods, then chances are you have a memory leak in the unmanaged space.

Determining where we are leaking

To find out where and what you are leaking you'll want to fire up a Memory Profiler, such as Ants Memory Profiler and start capture snapshots for comparison.  When comparing you'll want to view the class list and sort by the unmanaged space.
Class List view

Now click on the top offenders and drill down into your Instance Categorizer and start looking for your objects that are maintaining the references that are preventing cleanup.  You've now got a good place to start from an object/class perspective, which will give you a hint as to where you need to look.
Drilldown into Instance Categorizer

Drilldown into Instance Graph

At this point I would then open the object in Visual Studio and start reviewing the code, perhaps adding some break points and watching what happens to my memory profile as I pass through certain code parts while completing the necessary issue reproduction tests.   In general, since we are looking for a leak, it's best to put your breakpoints within the creation and dispose (cleanup) functions of your objects.  Once you break on a Dispose() call, look at the memory footprints before and after processing and see if anything has happened.  If the memory isn't getting cleaned up, chances are your Dispose is not properly handling things and needs to be altered.

Fixing the leak

Now that we've pinpointed where the problem lies we need to patch the leak.  In the simple case it may be just adding a Dispose() call to some object that isn't be called.  In other cases we may need to force a garbage collection - GC.Collect() - typically this is needed when you see a lot of objects sitting on the finalizer queue and not being cleaned up in a timely fashion.

Verify your fix

Congrats!  You've found and fixed the leak, now it's time to verify.  Fire up your memory profiler again and take a baseline snapshot, then conduct the steps to reproduce the issue and take another snapshot.  Does your virtual memory still appear to be leaking or have you plugged the hole?  If you
see that the hole is plugged, then congrats you're done!   If not, then rinse and repeat.

Fix Verified, unmanaged memory footprint reduced by 2x

Conclusion

Unlike Java where the GC is fairly good and doing it's job and therefore explicit System.GC calls are not recommended, it appears that C#/.NET does need and encourages developers to put in explicit collect calls.