"This is the official LOGbinder page for tracking the Exchange  24-hour mailbox audit bug. You can keep up with everything my team knows and  is doing by checking this page often.” – Randy Franklin Smith

LOGbinder bulletin, December 14, 2015 -- While investigating a support case, LOGbinder discovered a non-obvious yet critical bug in Exchange audit logging that essentially delays your ability to detect non-owner mailbox access for 24 hours. The PowerShell cmdlets New-MailboxAuditLogSearch and Search-MailboxAuditLog produce audit search results that are unpredictable and inconsistent when auditing all mailboxes and the start date is less than 24 hours ago.

We have notified Microsoft about the problem and they have confirmed it as a bug but have told us that they have no timeline for a bug fix. The bug affects Exchange Server 2010, 2013 and 2016.

What is the risk?

The risk to you is that you may never know you have an Exchange Server data breach – despite performing regular audits.

This strikes to the very core of application security audit. Not only is a 24-hour audit delay 24 hours too long, audit integrity is absolutely critical to security intelligence operations.

Details about the Bug

We encourage you to watch an 8-minute clip of our recent Exchange mailbox audit webinar embedded below. In this clip we discuss specifics about the bug and how it could be affecting you.

Here are the highlights about the bug:

• The bug returns unpredictable results when auditing all mailboxes: you may get no events at all when there are events, you may get only a few events, or you may get all matching events as you should. 
• Unless you are looking for specific events repeatedly – or you audit your audit results – you will never notice the problem. 
• The bug is not documented. We have reported this issue to Microsoft; they have confirmed it is a bug and said they have no solution timeline to share. Microsoft’s suggested workaround is to use a date range greater than 24 hours.

LOGbinder’s View on this Issue

The bug introduces a huge business, compliance and security impact. It is simply unacceptable to be unable to detect or respond to information theft for 24 hours. Security audits need to be available in seconds, not minutes! A delay brings compliance issues and prevents organizations from handling Edward Snowden-like information grabs before the culprit is out of reach.

We believe that you need to get audit results off the system (or application) where they are generated as fast as possible, without causing harm to the application or system while using least privilege.

What LOGbinder is Doing

Our development team is solving the problem. To ensure audit integrity we have released an update to our Exchange audit solution that all customers should download and begin using immediately. LOGbinder for Exchange 3.1 allows customers to choose whether they wish to perform audits in less than 24 hours, but defaults to the delay that we know will provide all the audit results requested. Click here to download: https://www.logbinder.com/Form/LBEXDownload

But LOGbinder for Exchange 3.1 is only the first phase of the ultimate solution. We are working with Microsoft and the Exchange Server community to raise awareness of the issue to get it to the top priority within Microsoft.

Exchange Server’s audit function is quite good. Leaving the bug aside, few applications make such an effort to audit events. Microsoft deserves a lot of credit (more than they usually get) for embedding both an admin and mailbox audit function in the application.

But if and until Microsoft does fix this bug we realize you need to protect your organization and fulfill compliance requirements.

Coming soon: Targeted, Synchronous Mailbox Audit Log Collection

Our new edition of LOGbinder for Exchange due Q1 2016 will continue to maintain audit log integrity using least-privilege and minimal impact, and deliver the admin audit as well as mailbox audit logs with a new robust and stable technology to provide audit logs for high-priority mailboxes in near real-time!

Like our current version, you can specify groups or OUs of executives or other sensitive mailboxes, and LOGbinder will use synchronous mailbox audit log searches on those groups and/or OUs. (To understand why “synchronous” is so significant, watch the full edition of our webinar Detect and monitor threats to your executive mailboxes with Exchange mailbox auditing. Non-targeted mailbox audit logs that should also be monitored for non-owner access will be returned in 24 hours (if and when Microsoft fixes the bug).

The benefit is that your targeted mailboxes will get to your SIEM in minutes if not seconds! Click here to get the beta of the newest edition of LOGbinder for Exchange when it becomes available.

What You Can Do

Stay up-to-date and get the latest innovations from LOGbinder.

If you are already a LOGbinder for Exchange customer, the first thing you should do is download the latest version. This will ensure you get all the audits you should be getting to the SIEM, even if they are delayed 24 hours. Some organizations have reported that they have no issues with the 24-hour delay.

Register for a beta of the coming new edition of LOGbinder for Exchange that will deliver targeted mailbox audit using synchronous search and real-time log delivery.

Open a support case with Microsoft to let them know this bug is a problem for you and send us the case ID. LOGbinder is taking a proactive approach with Microsoft and the Exchange Server community to help solve this problem and your participation will be of great value to the process.

Join the discussion at http://forum.ultimatewindowssecurity.com/Forum1608-1.aspx.

Bookmark this page and check it often to see what news and updates there have been. We will keep you up-to-date with everything we know and are doing by adding news items to the top of the page (content will be in reverse date-order top-to-bottom).

One of the main goals for LOGbinder SP 4.0 is to address the issue many of you have been experiencing with large memory consumption for the LOGbinder service. We received reports that memory consumption would sometimes rise to be over 10GB—and keep rising! Certainly this is not acceptable.

Before describing how LOGbinder SP 4.0 helps to address the problem, let me give a little background. LOGbinder’s interaction with SharePoint is mainly in these areas: (a) retrieve raw audit log data, (b) resolve IDs and other event data.

How does it interact with SharePoint? We use SharePoint’s server-side object model API. Although it would be more efficient to go directly to the underlying SQL database, license and support contracts prohibit accessing SharePoint data in this way. The server API has a dependency on the .NET Framework version 3.5 (for SharePoint 2013, it depends on Version 4.0).

So, when we started looking for the reason for LOGbinder’s large memory usage, we assumed it must be a memory leak in how we are using the SharePoint API. It’s been well documented that this API, if not used carefully, can result in memory leak. We didn’t find any.

What our development team did find is a weakness in the .NET Framework itself. A .NET application has two sections of memory, a stack for small objects, and a heap for objects larger than 80KB. As objects are created, they are added to one or the other of these locations—small objects to the stack, larger ones to the heap. For LOGbinder to do its “b” step, that is, resolving IDs and other event data, it has to access site collections, libraries, documents, etc. Many of these are small objects, but some larger objects are created that need to be added to the heap.

When no longer needed, LOGbinder will mark both large and small objects as obsolete, and the .NET Framework will clear out those locations in memory and reclaim the memory space.

It is in this large object heap where the .NET Framework has a problem—it will clear out the location in memory, but it does not always properly reclaim the space. Nor is .NET properly compacting the large object heap. Thus, as LOGbinder continues to run, it expands its memory ‘footprint,’ adding new data to the top—what we see is ever-expanding memory usage. In actuality, much of that memory space is empty. For example, if Windows reports that the LOGbinder service is using 10GB, it may be that only 2GB is actually being used.

You might be thinking, “I use .NET applications all the time, and I don’t see ‘runaway memory usage.’ So how come LOGbinder SP does?” The answer? The expanding memory problem shows up only in long-running processes (i.e. services) that consume large chunks of data. So you wouldn’t experience this with a desktop application, nor would you see this with a service that uses data objects where each is less than 80KB. LOGbinder SP routinely accesses objects bigger than 80KB.

In .NET 4.5, Microsoft has begun to address this problem (see http://blogs.msdn.com/b/dotnet/archive/2012/07/20/the-net-framework-4-5-includes-new-garbage-collector-enhancements-for-client-and-server-apps.aspx). But we're not in a position to use this, since SharePoint requires using specific .NET versions.

How does LOGbinder SP 4.0 address large memory consumption? The LOGbinder SP service will restart itself when memory consumption reaches a certain threshold (8GB), but in a seamless way so as not to impact the timing for processing site collections. Our support staff will be providing further details on how this works.

In conclusion, it is helpful to have a reality check: even with this new solution LOGbinder SP 4.0 will consume a large amount of memory. In order to provide value, it has to search through sites, libraries, and documents on the SharePoint farm. The larger and more complex the farm, the greater impact this will have on the service. Of course, we continue to look for ways to make this more efficient. We are proud of our product, and we are committed to making it better so that it serves your needs effectively.

We would love to hear your feedback! Let us know what you find in your own environment with the new LOGbinder SP. Drop us a line to support@logbinder.com and be sure to include a reference to LOGbinder SP 4.0 in the subject line.

From time to time, customers receive the following error in conjunction with LOGbinder, we've even had it in our lab environment. We are trying to get to the bottom of the issue, and we need your help--even if you haven't had the problem yourself.

"Unable to configure SharePoint export. Details: Cannot open database "WSS_Content" requested by the login. The login failed. Login failed for user 'SHAREPOINTSERVER\logbinderaccount'. SQL Database 'WSS_Content' on SQL Server instance 'SHAREPOINTSERVER\OfficeServers' not found. Additional error information from SQL Server is included below. Cannot open database "WSS_Content" requested by the login. The login failed. Login failed for user 'SHAREPOINTSERVER\logbinderaccount'."

Through our research so far, we find it is a common error with SharePoint, occurring most often (a) in conjunction with an upgrade or (b) when SharePoint is accessed via a console application or service. The second scenario applies to LOGbinder SP. And this error occurs even though the account is a SharePoint farm administrator, with all privileges being assigned correctly.

The workaround is to add the account manually as a user to the SQL database WSS_Content, assigning permissions as database owner (db_owner database role). This works most of the time. However, we would like to get some more definitive answers on this issue. To do that, could you answer the following questions?

If you have had the error:

1. Did it occur in your test or production environment?
2. What did you do to work around the error?

If you have NOT had the error:

1. For the account you use to run the LOGbinder agent, what SharePoint groups does it belong to?
2. Is the account a domain account or a local user account?
3. To what domain groups does it belong?
4. What local groups?
5. If you have a multi-server SharePoint farm, from which server is LOgbinder running? (database, front-end, application, etc.)

You may send your response to support@logbinder.com. Any information you provide will be kept strictly confidential. Thank you.

You will happy to know that we have released a new version of LOGbinder SP, version 3.1.  Here's what we've updated:

• During the trial period, output to the Security Log is now permitted 
• New event #62 “Document fragment updated”: This event handles the SharePoint event type “FileFragmentWrite”
• LOGbinder will now reset its connection automatically to the site collection(s) every 23 hours, which will help eliminate authentication timeouts and reduce resource consumption
• Also, a number of bug fixes and performance improvements

You can download the latest version at http://www.logbinder.com/form.aspx?action=download 

We are pleased to announce the release of LOGbinder SP 3.0. The fundamentals of LOGbinder remain the same. It continues to support SharePoint 2007 and SharePoint 2010, Foundation (WSS), Standard, and Enterprise editions. While nothing was removed from LOGbinder, we have added a number of features that will make your use of LOGbinder even more effective.

For customers with active LOGbinder SP support contracts, please watch for an important email that will give you important upgrade information.

What’s New

·       Farm-Wide Default Audit Policy –  Have you every wished you could just configure audit policy once for the whole farm?  Now you can!  You can now define a default audit policy and apply it to all existing site collections.  Better yet…

·       New site collections automatically audited – You no longer need to remember to enable auditing on new site collections.   The new default audit policy is automatically applied to new site collections.  This will be a real boon to farms with self-service site collection enabled.

·       Audit Trail Integrity - Automatic detection and response to tampering

Comprehensive Description of Enhancements

Default Audit Policy

LOGbinder SP allows you to specify a “Default Audit Policy.” This allows you to define a policy once, and apply it to the site collections you specify—thus allowing for more consistency in audit settings.

The default audit policy can also be applied to new site collections. Thus, soon after a site collection is created, LOGbinder SP can automatically enable auditing and begin processing its audit logs.

Configure Inputs (SharePoint Site Collections)

There are now three ways to specify the audit settings for a site collection:

·         “Custom audit policy”: Corresponds to Version 2.x, where you use LOGbinder to set the site collection’s audit policy.

·         “Allow Site Collection Administrator to configure audit policy using SharePoint’s administration page”: This allows you to set the audit policy in SharePoint (if running the Microsoft SharePoint Server edition).

·         “Use LOGbinder’s default audit policy”: Once the default audit policy has been defined, it can be applied to the site collection. If the default audit policy later changes, LOGbinder will automatically apply the changes to the site collection’s audit settings.

Configure Outputs

There is no change to functionality, but the text of the properties window has been updated to make the options more clear.

SharePoint Farm Properties

Available from Options and from the Configure Inputs windows, the “SharePoint Farm Properties” window displays basic information about the SharePoint farm—the number and names of servers, farm ID, and edition information. Some of this information is needed when purchasing a LOGbinder license.

Tamper Warnings

While LOGbinder is processing events, it will perform actions that generate SharePoint events. These same actions, if performed maliciously by a SharePoint user, could compromise the integrity of the audit trail. In order to distinguish between authorized and unauthorized changes, when LOGbinder processes these events, it will indicate it performed the action—or whether the action might be unauthorized. A tamper warning may be generated by these events:

·         Audit policy change: When processing event #11 “Site collection audit policy changed” or #12 “Audit policy changed,” LOGbinder will determine if the change overrides the setting in LOGbinder. If so, LOGbinder will reset the audit policy and generate a tamper warning.

·         Audit logs deleted: When processing event #20 “Audit logs deleted,” LOGbinder will determine whether LOGbinder deleted the logs. An additional line is added to this event “Purge performed by LOGbinder”—the value will be “Yes” if LOGbinder performed the purge. If not, a tamper warning will be generated.

An additional event, #60 “Possible tampering warning” has been added to the LOGbinder SP event list.

LOGbinder Diagnostic Events

The diagnostic events generated by LOGbinder can be accessed from within LOGbinder. These events are still written to the Application event log, but by selecting the view “LOGbinder Events”—or one of the subviews—you can view the diagnostic events right from within LOGbinder.

Reports

The new “Reports” contain our recommended SIEM reports, where you can inspect LOGbinder’s results right from within LOGbinder. However, LOGbinder is still best used in conjunction with your SIEM solution—and LOGbinder now provides basic information on integrating your SIEM solution. In the meantime, however, you can use the basic reports within LOGbinder to view the filtered LOGbinder events.

Privileges

The user account you use to install and configure LOGbinder is required to be a member of the local administrators group. This was recommended in Version 2.x, but now is required.

Licensing

LOGbinder SP 3.0 licensing is per farm—based on the number of front-end and application servers on the farm. The “Installation Code” will be based on the ID of your SharePoint farm instead of the fingerprint of the server on which LOGbinder is installed. This means that if you install LOGbinder on a different server on the farm, the same license key will be used for both. Also, if you upgrade the Windows server, you will not need to obtain a new license key.

LOGbinder SP 3.0 has only two editions: “Microsoft SharePoint Foundation 2010” and “Microsoft SharePoint Server 2010.” (For SharePoint 2007, “Microsoft Windows SharePoint Services 3.0” and “Microsoft Office SharePoint Server 2007.)

The server count is now included in the licensing window. In addition, you can obtain the server count through the menu File\Options, then click on “SharePoint Farm Properties,” and note the value of the “Number of servers requiring LOGbinder license.”