One of the main goals for LOGbinder SP 4.0 is to address the
issue many of you have been experiencing with large memory consumption for the
LOGbinder service. We received reports that memory consumption would sometimes rise
to be over 10GB—and keep rising! Certainly this is not acceptable.
Before describing how LOGbinder SP 4.0 helps to address the
problem, let me give a little background. LOGbinder’s interaction with
SharePoint is mainly in these areas: (a) retrieve raw audit log data, (b)
resolve IDs and other event data.
How does it interact with SharePoint? We use SharePoint’s
server-side object model API. Although it would be more efficient to go
directly to the underlying SQL database, license and support contracts prohibit
accessing SharePoint data in this way. The server API has a dependency on the
.NET Framework version 3.5 (for SharePoint 2013, it depends on Version 4.0).
So, when we started looking for the reason for LOGbinder’s
large memory usage, we assumed it must be a memory leak in how we are using the
SharePoint API. It’s been well documented that this API, if not used carefully,
can result in memory leak. We didn’t find any.
What our development team did find is a weakness in the .NET
Framework itself. A .NET application has two sections of memory, a stack for
small objects, and a heap for objects larger than 80KB. As objects are created,
they are added to one or the other of these locations—small objects to the
stack, larger ones to the heap. For LOGbinder to do its “b” step, that is,
resolving IDs and other event data, it has to access site collections,
libraries, documents, etc. Many of these are small objects, but some larger
objects are created that need to be added to the heap.
When no longer needed, LOGbinder will mark both large and
small objects as obsolete, and the .NET Framework will clear out those
locations in memory and reclaim the memory space.
It is in this large object heap where the .NET Framework has
a problem—it will clear out the location in memory, but it does not always
properly reclaim the space. Nor is .NET properly compacting the large object
heap. Thus, as LOGbinder continues to run, it expands its memory ‘footprint,’
adding new data to the top—what we see is ever-expanding memory usage. In
actuality, much of that memory space is empty. For example, if Windows reports
that the LOGbinder service is using 10GB, it may be that only 2GB is actually
You might be thinking, “I use .NET applications all the
time, and I don’t see ‘runaway memory usage.’ So how come LOGbinder SP does?”
The answer? The expanding memory problem shows up only in long-running
processes (i.e. services) that consume large chunks of data. So you wouldn’t
experience this with a desktop application, nor would you see this with a
service that uses data objects where each is less than 80KB. LOGbinder SP
routinely accesses objects bigger than 80KB.
In .NET 4.5, Microsoft has begun to address this problem
But we're not in a position to use this, since SharePoint requires using
specific .NET versions.
How does LOGbinder SP 4.0 address large memory consumption?
The LOGbinder SP service will restart itself when memory consumption reaches a
certain threshold (8GB), but in a seamless way so as not to impact the timing
for processing site collections. Our support staff will be providing further
details on how this works.
In conclusion, it is helpful to have a reality check: even
with this new solution LOGbinder SP 4.0 will consume a large amount of memory.
In order to provide value, it has to search through sites, libraries, and
documents on the SharePoint farm. The larger and more complex the farm, the
greater impact this will have on the service. Of course, we continue to look
for ways to make this more efficient. We are proud of our product, and we are
committed to making it better so that it serves your needs effectively.
We would love to hear your feedback! Let us know what you
find in your own environment with the new LOGbinder SP. Drop us a line to firstname.lastname@example.org and be sure to
include a reference to LOGbinder SP 4.0 in the subject line.