LOGbinder Blog

Updates, Tips and News   RSS Feed  

Today we revolutionize using Windows Event Collection at scale

Fri, 04 Sep 2020 15:53:09 GMT

What we are announcing today with Supercharger for Windows Event Collection reminds me of how far technology has come.  I remember being so excited to sit down in front of the great Commodore 64 thinking how amazing this is.  Now almost 40 years later and my processor has as many cores as that C64 had bits.  Why am I talking about 40-year-old computers?  Back then I, like many, thought that the C64 was an amazing machine. But look at what we have today just 40 years later.

Today, Windows Event Collection/Forwarding (WEC/WEF) is becoming a well-known well-functioning technology.  It’s been around since Vista and Server 2008 and it’s a great technology.  What makes WEC an awesome technology?  To start, it’s built-in to Windows and has no agents, no polling, no noise and can be centrally managed.  In the past 12 years nothing much has changed with WEC.

Now that WEC is being implemented more in the real world it’s becoming easier to see where it is lacking.  For example, once you start to scale out with multiple collectors and thousands (or tens or hundreds of thousands) of forwarders, then your collectors can become unstable or even unusable.  The solution?  Setup more collectors.  Create the same subscription on each collector but assign different computers to each collector.  Good idea, right?  In theory, yes.  Soon it will become obvious that keeping collectors and subscriptions consistent is a full-time job.  The next thought may be to just use AD groups.  But then you run into the issue of new computers and decommissioned computers.  That issue is nothing compared to the fact that group membership changes don’t take affect until either a system reboot or purging of the Kerberos ticket.  I think it’s safe to say that for most, if not, all of us, we just can’t do a mass reboot of servers/workstations.   So that leaves the option of purging the Kerberos ticket.  Now we have to bring in technologies like System Center, PS Remoting or tasks in our GPO’s.  There has to be a better way.

So, let’s nix the idea of groups in the WEC subscriptions.  Can’t we just add individual computers to the subscription? 

Yes, you can but as it turns out WEC’s subscription memory structure is limited to about 1,800 allowed computers.  Many organizations regularly need to assign tens of thousands of forwarders to a single collector. So now that’s going to require multiple duplicate subscriptions targeting unique sets of 1,800 (1,500 to be safe) forwarders each. 

 

Tired yet???  Just thinking of the management of this setup gets me stressed.  All this makes two points very clear.  WEC is a great foundation for an enterprise logging pipeline but

  1. it needs care and feeding
  2. becomes unwieldly on its own especially once you start scaling out

So, since its release 12 years ago not much has improved WEC, that is until now!

Back in 2018 we released Supercharger for WEC.  One could say it was the Commodore 64 of its time.  It had what we called Distributed Subscriptions.  Supercharger made use of a dedicated OU in AD where it managed groups that were used to balance subscriptions across multiple collectors.  Did this fix the issue that many WEC environments were facing with large numbers of endpoints?  Yes, but again we have the issue of computers not seeing that they are added to groups until reboot.  So, Supercharger was constantly waiting for load balance maintenance to take effect.

Now, on Sept 3, 2020 we released a new version of Supercharger.  We have redesigned distributing subscriptions with our new Load Balanced 2.0 technology.  

This revolutionary enhancement means no more waiting for endpoints to see group additions.  This also means Supercharger no longer requires a dedicated OU in AD and a dedicated service account with permissions to create/modify groups in that OU.  You may be thinking, “hey what about WEC’s limit of 1,800 forwarders to subscription limit?”  We solved that by programmatically creating multiple WEC subscriptions with 1,500 computers assigned to each one.  These individual subscriptions are known as “shard sets” in Supercharger.

Now load balanced subscriptions take effect immediately.  If you need to add, remove, or replace a collector this can be completed instantaneously.  We are not exaggerating when we say that the reaction time has been reduced from weeks to seconds!

With Supercharger, not only do you get one pane of glass visibility into your WEC environment but you no longer need to “jump box” around from collector to collector to monitor subscriptions.  So, you can escape being tied down to RDP and Event Viewer when managing WEC.  Also, Microsoft has admitted that at a certain point, once you reach X number of subscriptions and forwarders, Event Viewer just gets overloaded and will stop working.  Obviously, this makes it impossible to manage subscriptions in some environments.  Add on to this the health change alerting, performance monitoring, trend analysis, policy-based control and load balancing 2.0 and it’s clear that Supercharger is a must have for any WEC implementation. 

Download Supercharger today and see just how easy a huge implementation of WEC can be.  Just imagine having all of your Windows endpoints send event logs to a collector in under 15 minutes.  With Supercharger we’ve made the impossible possible. 


All LOGbinder products updated

Tue, 10 Dec 2019 17:18:50 GMT

Almost 12 years ago, my first LOGbinder product (LOGbinder for SharePoint) was created.  Since then we've developed software to help you audit SQL Server and Exchange admin and mailbox audit logs.  With the advent of our latest product, Supercharger for Windows Event Collection, we are now one of the biggest resources for the deployment, implementation and troubleshooting of Windows Event Collection (WEC).  Recently we released updates to all four of our products.  What's new?  At the bottom of this email are just a few of many new features and enhancements to our product line.  

I realize that a bulleted list of "features" may not seem that impressive, so I invite you to download any or all of our products and test them for yourself to see how they can help you audit the security actions in your environment.  For example, do you want to set a custom audit policy for every single one of your SharePoint sites including new sites that get create and then also get alerted if a malicious actor changes that audit policy?  Then try LOGbinder for SharePoint.  Do you want to audit SQL Server audits without touching the SQL Server or DB's once the audit is created?  Your SQL admins would love for you to try out LOGbinder for SQL Server.  Do you want to collect any log in event viewer from every workstation and server in your domain?  If your SIEM's cost is based on EPS or data storage, then Supercharger may pay for itself by allowing you to leave the noise at the source.

You can click the product to see all the latest changes:

  • Supercharger for WEC 19.10
    • Reports added
      • Comprehensive forwarder analysis - see every possible detail about every forwarder in your domain.  Excellent resource for troubleshooting problem forwarders
      • Collector performance history - see trends and patterns about collectors EPS and CPU.  Helpful for monitoring collector performance and resource planning
    • Maintenance button added to subscriptions of load balanced distributed subscriptions so you can maintain them on demand
    • Enhanced custom event log creation
  • LOGbinder for SharePoint 7.0.1
    • Filter events based on site
    • Error handling improved to make the service more resilient
    • Performance enhancements to speed up processing
    • Noise filtering 
    • Support for the latest versions of SharePoint
  • LOGbinder for SQL Server 5.0.1
    • Enhanced error handling
  • LOGbinder for Exchange 4.0.1
    • Redesign of mailbox audit configuration wizard
    • Coded workarounds for the "Too many audit requests" Exchange issue
    • Performance enhancements to speed up processing
    • "Apply Now" option for instantly applying the audit wizard configuration​

If you're already familiar with WEC or just learning, you'll want to view Randy Franklin Smith's recent webinar on WECBuilding a Resilient Logging Pipeline: Windows Event Collection Tips and Tricks for When You Are Serious About Log Collection.

Get instant pricing for Supercharger and our LOGbinder for SharePoint/SQL/Exchange products here:  Instant Quotes  

Over the past few months we've been listening to you.  Most of the enhancements and bug fixes in our latest releases are because of you.  The feedback and suggestions on our forum and support portal have helped us continue to improve our products.

If you are already a licensed user of our products and have a current support contract, then upgrading is easy.  Just find the product you need to upgrade on our download page.  Download the installer you need and just install on top of your current installation.  You will most likely need to request an updated product key at support.logbinder.com.  If you are upgrading Supercharger you just need to upgrade the manager.  All the collectors will upgrade themselves.

Thanks again for your support and I look forward to your feedback.

Randy Franklin Smith


How to analyze where events are coming from and how many

Thu, 31 Oct 2019 15:41:08 GMT

Recently we had an issue with a Supercharger customer/  They had 40+ distributed subscriptions and two of their many collectors were having latency issues. After hours and hours of investigation, research and troubleshooting we identified the issue. Of their tens of thousands of forwarders, just a few (yes not few thousand or few hundred but a handful) were generating a huge amount of events in comparison to the rest. This resulted in an unbalanced load balanced subscription. As a result of this troubleshooting, our Event Count By Source utility was born. Just download the utility and run it on a collector against some of the logs. You will see how it totals events by source and sorts them in descending order.

This utility is intended to help you determine where forwarded events are coming from and in what quantity. It analyzes the specified log and counts events by source computer and source log. 3 tab delimited files are produced in the current directory named after the type of data, log name, and computer name allowing you to co-mingle output from multiple executions of the program.

The 3 files summarize event counts by source machine only, source log only and by log and machine. A 4th file simply documents the number of events in the log.

There is only one parameter - log name. For a list of logs run "get-winevent -listlog *".
Omit log name and it will default to ForwardedEvents.

Download Event Count By Source utility.

This is a free utility from LOGbinder (www.logbinder.com) which is a division of Monterey Technology Group, Inc.
(c) 2019 Monterey Technology Group, Inc (MTG). You are free to use and copy this program for lawful uses. Use at your own risk. By your use of this program you hold MTG harmless of any untoward results.


Kerbpurge - Group memebership without a reboot

Sat, 01 Jun 2019 11:30:18 GMT

KerbPurge - What is it?

When you add a computer to a group in Active Directory, the computer does not know that it has been added to the group until a reboot happens. There are many obvious reasons why this is a problem. For example, you can't just reboot production servers at any time. Most organizations have some sort process in place for scheduling server reboots which in itself can be a time consuming process. When it comes to Windows Event Collection there are many reasons for adding endpoint forwarders to groups, especially if you are using Supercharger. For example, Superchargers builtin load balanced or distributed subscription feature rely's on group changes to keep forwarders balanced across the number of specified collectors. This is why Randy Franklin Smith of UltimateITSecurity.com designed and wrote KerbPurge.

Benefits and Features

  • Safely and efficiently make Windows computers see group membership changes
  • Tiny Windows service
  • Installable via Group Policy's Software Installation feature
  • Only purges tickets for the Network Service logon session and only when group membership has been changed for a computer
  • No measurable resource usage

Installation and Configuration

To install and configure Kerbpurge visit our knowledgebase for the step-by-step guide.

    Least Privilege Workaround for SQL DB Access

    Sat, 08 Dec 2018 10:21:56 GMT
    In the past we have explained how LOGbinder for SharePoint uses SQL privileges. We also informed you about the unfortunate workaround of giving dbo access to certain DB's in SQL in the sporadic cases when the SharePoint API interferes with access to the databases. 

    This was never a "workaround" that we were really happy with.  Giving dbo access is not only like giving the bank the title to your home as collateral for the mortgage but also giving them a letter that says "Stop by anytime you want and while you're here feel free to repaint the walls and help yourself to the scotch in the pantry."

    Thankfully, we have found a proper workaround that does not require dbo access.  There is a role on the SharePoint SQL DB's named "SPDataAccess".  We have found that giving the service account this role grants enough access for LOGbinder for SharePoint to function properly.  Again we would like to specify that this is not the standard configuration needed with LOGbinder for SharePoint.  This is only used in the rare situations when the SharePoint API is giving issues with DB access.  For most of our customers the permissions set within SharePoint itself for the service account is all that is needed. 

    There are two ways to give the service account this role.  One is using the SharePoint Management Shell and the other is directly in SQL (in our example below using SSMS). 

    Our preferred method is making the changes directly in SQL.  We noticed that when using the SP Management Shell an extra role is given.  We also noticed that this is not always the case as well.  Sometimes the extra role is given and sometimes it is not.  Why?  We don't know.  Maybe it's a hidden Microsoft feature.

    Here is how to make the changes using SSMS.

    1. In SSMS add your service account as a login.


    2. Open the logins properties and locate the three databases that your SharePoint farm is using for the Admin Content, Configuartion and WSS Content databases..  In this instance we have SharePoint_AdminContent(GUID), SharePoint_Config2019 and WSS_Content(GUID). 


    3. For each database map the SPDataAccess role to the login.  You will notice that for the WSS_Content db, after saving the role change SSMS also grants the PSDataAccess and the PSReportingSchemaAdmin role.  If you have more than one content db, then you will have to perform these steps on all applicable db's with the WSS_Content prefix.  For more information on how to set SPDataAccess on a large number of content databases, click here.


    You can also perform the steps above with a simple cmdlet using the SharePoint Management Shell.  Run the following cmdlet:

    Get-SPContentDatabase | Add-SPShellAdmin -UserName domain\ServiceAccount

    So in our example below we ran "Get-SPContentDatabase | Add-SPShellAdmin -UserName lab\sp2019srvacct".  Notice that doing this grants an additional role on all three databases; the SharePoint_Shell_Access role. As security experts our recommendation is obviously whichever process results in the least privilege needed to get the job done which, in this case, is making the changes via SSMS.


    What does the SPDataAccess role allow?  According to TechNet, the SP_DATA_ACCESS role will have the following permissions:

    • Grant EXECUTE or SELECT on all SharePoint stored procedures and functions
    • Grant SELECT on all SharePoint tables
    • Grant EXECUTE on User-defined type where schema is dbo
    • Grant INSERT on AllUserDataJunctions table
    • Grant UPDATE on Sites view
    • Grant UPDATE on UserData view
    • Grant UPDATE on AllUserData table
    • Grant INSERT and DELETE on NameValuePair tables
    • Grant create table permission
    Reference:  TechNet

    previous | next

    powered by Bloget™