One of the main goals for LOGbinder SP 4.0 is to address the issue many of you have been experiencing with large memory consumption for the LOGbinder service. We received reports that memory consumption would sometimes rise to be over 10GB—and keep rising! Certainly this is not acceptable.

Before describing how LOGbinder SP 4.0 helps to address the problem, let me give a little background. LOGbinder’s interaction with SharePoint is mainly in these areas: (a) retrieve raw audit log data, (b) resolve IDs and other event data.

How does it interact with SharePoint? We use SharePoint’s server-side object model API. Although it would be more efficient to go directly to the underlying SQL database, license and support contracts prohibit accessing SharePoint data in this way. The server API has a dependency on the .NET Framework version 3.5 (for SharePoint 2013, it depends on Version 4.0).

So, when we started looking for the reason for LOGbinder’s large memory usage, we assumed it must be a memory leak in how we are using the SharePoint API. It’s been well documented that this API, if not used carefully, can result in memory leak. We didn’t find any.

What our development team did find is a weakness in the .NET Framework itself. A .NET application has two sections of memory, a stack for small objects, and a heap for objects larger than 80KB. As objects are created, they are added to one or the other of these locations—small objects to the stack, larger ones to the heap. For LOGbinder to do its “b” step, that is, resolving IDs and other event data, it has to access site collections, libraries, documents, etc. Many of these are small objects, but some larger objects are created that need to be added to the heap.

When no longer needed, LOGbinder will mark both large and small objects as obsolete, and the .NET Framework will clear out those locations in memory and reclaim the memory space.

It is in this large object heap where the .NET Framework has a problem—it will clear out the location in memory, but it does not always properly reclaim the space. Nor is .NET properly compacting the large object heap. Thus, as LOGbinder continues to run, it expands its memory ‘footprint,’ adding new data to the top—what we see is ever-expanding memory usage. In actuality, much of that memory space is empty. For example, if Windows reports that the LOGbinder service is using 10GB, it may be that only 2GB is actually being used.

You might be thinking, “I use .NET applications all the time, and I don’t see ‘runaway memory usage.’ So how come LOGbinder SP does?” The answer? The expanding memory problem shows up only in long-running processes (i.e. services) that consume large chunks of data. So you wouldn’t experience this with a desktop application, nor would you see this with a service that uses data objects where each is less than 80KB. LOGbinder SP routinely accesses objects bigger than 80KB.

In .NET 4.5, Microsoft has begun to address this problem (see http://blogs.msdn.com/b/dotnet/archive/2012/07/20/the-net-framework-4-5-includes-new-garbage-collector-enhancements-for-client-and-server-apps.aspx). But we're not in a position to use this, since SharePoint requires using specific .NET versions.

How does LOGbinder SP 4.0 address large memory consumption? The LOGbinder SP service will restart itself when memory consumption reaches a certain threshold (8GB), but in a seamless way so as not to impact the timing for processing site collections. Our support staff will be providing further details on how this works.

In conclusion, it is helpful to have a reality check: even with this new solution LOGbinder SP 4.0 will consume a large amount of memory. In order to provide value, it has to search through sites, libraries, and documents on the SharePoint farm. The larger and more complex the farm, the greater impact this will have on the service. Of course, we continue to look for ways to make this more efficient. We are proud of our product, and we are committed to making it better so that it serves your needs effectively.

We would love to hear your feedback! Let us know what you find in your own environment with the new LOGbinder SP. Drop us a line to support@logbinder.com and be sure to include a reference to LOGbinder SP 4.0 in the subject line.