***This blog post is still important but outdated.  Please see this post for updated least privilege changes.***

Issue

In the blog on www.logbinder.com (Workaround if LOGbinder SP is having SQL database issues), a suggested workaround for insufficient privileges to SharePoint’s SQL databases is to add the LOGbinder service account as a database administrator (DBO). The question arises: How does LOGbinder SP use these elevated privileges?

Access to SharePoint databases

First, it must be understood that LOGbinder SP does not access SharePoint’s SQL databases directly. All access to SharePoint data is through the SharePoint Server Object Model (see http://msdn.microsoft.com/en-us/library/jj164060.aspx). LOGbinder SP does not execute any Transact-SQL commands directly, nor does LOGbinder SP access the SQL database directly to adjust database structure, privileges, and so forth.

The workaround suggested in the above blog is recommended based on troubleshooting in our labs, to address what apparently is a defect in the SharePoint Server Object Model. LOGbinder SP does not then use these elevated privileges to perform other activity.

LOGbinder SP’s use of SharePoint data

Even though LOGbinder SP accesses SharePoint through its object model, a secondary question may be: What activity does LOGbinder SP perform in SharePoint? LOGbinder SP’s main activity is to read SharePoint audit logs, as well as to read metadata about SharePoint site collection, lists, libraries, users, groups, and similar entities.

Through the SharePoint Server Object Model, LOGbinder SP does make some changes to SharePoint (the customer specifies these changes in the LOGbinder Control Panel). The changes LOGbinder SP will make to SharePoint include: adding/removing site collection administrators, adjusting audit policy settings for a site collection, adjusting the audit log trimming setting for a site collection, and deleting audit log records. (The documentation for LOGbinder SP contains details on these actions.) So, other than purging old log data and setting audit policy according to configuration settings by the administrator, there is nothing that LOGbinder does that modifies or could corrupt SharePoint content or the SQL database.

One of the main goals for LOGbinder SP 4.0 is to address the issue many of you have been experiencing with large memory consumption for the LOGbinder service. We received reports that memory consumption would sometimes rise to be over 10GB—and keep rising! Certainly this is not acceptable.

Before describing how LOGbinder SP 4.0 helps to address the problem, let me give a little background. LOGbinder’s interaction with SharePoint is mainly in these areas: (a) retrieve raw audit log data, (b) resolve IDs and other event data.

How does it interact with SharePoint? We use SharePoint’s server-side object model API. Although it would be more efficient to go directly to the underlying SQL database, license and support contracts prohibit accessing SharePoint data in this way. The server API has a dependency on the .NET Framework version 3.5 (for SharePoint 2013, it depends on Version 4.0).

So, when we started looking for the reason for LOGbinder’s large memory usage, we assumed it must be a memory leak in how we are using the SharePoint API. It’s been well documented that this API, if not used carefully, can result in memory leak. We didn’t find any.

What our development team did find is a weakness in the .NET Framework itself. A .NET application has two sections of memory, a stack for small objects, and a heap for objects larger than 80KB. As objects are created, they are added to one or the other of these locations—small objects to the stack, larger ones to the heap. For LOGbinder to do its “b” step, that is, resolving IDs and other event data, it has to access site collections, libraries, documents, etc. Many of these are small objects, but some larger objects are created that need to be added to the heap.

When no longer needed, LOGbinder will mark both large and small objects as obsolete, and the .NET Framework will clear out those locations in memory and reclaim the memory space.

It is in this large object heap where the .NET Framework has a problem—it will clear out the location in memory, but it does not always properly reclaim the space. Nor is .NET properly compacting the large object heap. Thus, as LOGbinder continues to run, it expands its memory ‘footprint,’ adding new data to the top—what we see is ever-expanding memory usage. In actuality, much of that memory space is empty. For example, if Windows reports that the LOGbinder service is using 10GB, it may be that only 2GB is actually being used.

You might be thinking, “I use .NET applications all the time, and I don’t see ‘runaway memory usage.’ So how come LOGbinder SP does?” The answer? The expanding memory problem shows up only in long-running processes (i.e. services) that consume large chunks of data. So you wouldn’t experience this with a desktop application, nor would you see this with a service that uses data objects where each is less than 80KB. LOGbinder SP routinely accesses objects bigger than 80KB.

In .NET 4.5, Microsoft has begun to address this problem (see http://blogs.msdn.com/b/dotnet/archive/2012/07/20/the-net-framework-4-5-includes-new-garbage-collector-enhancements-for-client-and-server-apps.aspx). But we're not in a position to use this, since SharePoint requires using specific .NET versions.

How does LOGbinder SP 4.0 address large memory consumption? The LOGbinder SP service will restart itself when memory consumption reaches a certain threshold (8GB), but in a seamless way so as not to impact the timing for processing site collections. Our support staff will be providing further details on how this works.

In conclusion, it is helpful to have a reality check: even with this new solution LOGbinder SP 4.0 will consume a large amount of memory. In order to provide value, it has to search through sites, libraries, and documents on the SharePoint farm. The larger and more complex the farm, the greater impact this will have on the service. Of course, we continue to look for ways to make this more efficient. We are proud of our product, and we are committed to making it better so that it serves your needs effectively.

We would love to hear your feedback! Let us know what you find in your own environment with the new LOGbinder SP. Drop us a line to support@logbinder.com and be sure to include a reference to LOGbinder SP 4.0 in the subject line.

Click here to get a copy of Randy Franklin Smith's new whitepaper: Top 6 Security Events to Audit in SharePoint.

You can find other SharePoint whitepapers at on our Resources page at LOGbinder.com.