Log Blind Spots – A Review of Cases Where System Logs are Insufficient

If you spend a few minutes browsing the websites of Log Management and SIEM tool vendors, you might come away with the conclusion that all your system audit and compliance problems are solved. Unfortunately, this rosy picture seems to ignore the ever-present problem of blind spots in audit reports: If your apps don’t log it, your audit report won’t show it.
Audit report tools may do a good job at interpreting and presenting log info, but we can no longer overlook two key facts:
1. Hundreds of critical security event types are not logged at all
2. Those events that are logged typically do not show what was done. Instead, the logs only show obscure technical details of the resulting system changes.

In this whitepaper, I’ll highlight a number of examples where these blind spots occur. In so doing, I’ll show a number of very common and basic system activities that one might think should generate auditable log entries, but in actuality they do not.
These examples include:

• Adding and Deleting IP Address
• Setting a Service to run as administrator
• Change Web server config file
• Change port usage for an active service

 
One possible way to eliminate blind spots is to implement custom log utilities, security platforms or WMI-based tools.  But to do this, the burden remains on you to know what you are looking for.  For the examples listed above, adding an IP Address change monitor won’t help with web config file changes, and vice verse. And more importantly, adding 4 different monitors for each of those issues won’t help capture the hundreds of actions that you’ll never be able to predict.  As the well-worn yet valuable expression states, “Expect the unexpected”.
User Activity Monitoring follows through on this philosophy. In the context of IT audit logs, perhaps the best way to expect the unexpected is to drop the paradigm of listing the actions that should be logged, and instead  simply monitor all user user actions.
Note: For the purpose of this article I’ve enabled FULL SYSTEM auditing on a Windows Server 2003 machine, which includes ALL the possible options that are being made available through Group Policy. In addition, I’ve installed an ObserveIT Agent on the machine, for the purpose of visually auditing the users’ actions.

 
 

Scenario 1: Changing a System’s IP Address

What the User Did: A privileged user logged onto a Win 2003 Server (via RDP in this case, but for the sake of discussion it could be any local or remote connection protocol). After logging in, this admin user opened the Advanced TCP/IP Settings (via Start > Settings > Network Connections > Local Area Connection > Properties > Internet Protocol > Properties > Advanced). Once there, he removed an IP address (10.1.200.178), and then added a different IP address (10.1.200.179).

Advanced TCP/IP Settings
Security and Audit Implications of this Action: Adding and IP address might allow bypassing of firewall settings and may also interfere with proper execution of critical services.
What shows up in system event logs: With full auditing enabled, a total over 11,000 log events were triggered during the 30 seconds it took the user to delete and add an IP address. Almost all the log entries were of “Object Access” category. Searching within the logs for the terms “TCP”, “IP” or “179” (last 3 digits of the IP address added) brought back numerous search, but all were false hits.  (ex: “IP” appears in the filename “wshtcpip.dll” within one log entry, another log entry having Operation ID “74312179”.) No log entry refers explicitly to the action taken.
It may be possible for a highly-trained system security expert to piece together the log entries and determine what actions took place. But it would involve a time-intensive forensic analysis by a sparse and expensive resource. Do you have highly-trained security experts that are bored with nothing better to do than piece together log entries?

Event Viewer: 11,000 log entries in 30 seconds, dozens of false hits, no clear picture
What User Activity Monitoring shows you: A user-oriented textual audit log shows that “brad” logged on as “administrator”, and the list of actions tells the story of what he did: Network Connections > Properties > TCP/IP Properties > TCP/IP Address. This already is much more than information than what is accessible in the system logs.  Adding video replay of the session then shows even more details.


ObserveIT Audit Log: A Table of Contents of the user session


ObserveIT video replay of user changing the IP Address

Scenario 2: Setting a Service to run as administrator

What the User Did: An admin user changed the properties of a Service (via Start > Settings > Control Panel > Administrative Tools > Services). Once there, he selected the “Cryptographic Services” service and marked it to run as administrator.
Note: Forget the question whether this is something someone would want to do, or not. That’s not the point. The point is that is someone performs such an action, their action will not be identified.


Run a service as Administrator
Security and Audit Implications of this Action: Enabling a service that is not secure to run as administrator can enable remote hacking and can cause the service improperly affect sensitive system configuration and data.
What shows up in system event logs: Over 24,000 log events were triggered during the 40 seconds it took the user to change the Run As credentials. Despite the sheer volume, no log entries included the word “Crpytographic” (the name of the service)!
Again, a full-throttle investigation by system experts might unearth the true actions, but this task makes biblical archaeology look easy.

Event Viewer: 24,000 log entries in 40 seconds,
no indication of the Service that was modified
What User Activity Monitoring shows you: As in the previous example, ObserveIT shows a clear chronological timeline of what the user actually did: open Control Panel and then go to Cryptographic Services Properties. And again, video replay shows even more.

ObserveIT Audit Log

Video replay of Service Run As credential
 

Scenario 3:Change web.config  (IIS webserver configuration file)

What the User Did: Via Windows Explorer and Notepad, the user made a simple change to an XML attribute in the file “web.config”, changing a ‘0” (false) value to “1” (true).

Editing web.config with Notepad
Security and Audit Implications of this Action: Changes to this file will affect how the web server runs, in numerous different ways. This can expose security risks, and can also affect proper operations.
What shows up in system event logs:  6,000 log entries cover the 20 seconds it took to make the change. One log entry indicates that “Notepad” was launched. Another log entry indicates that “web.config” was added to the “Recent Files” list in Windows.  A third log entry seems to show (not convincingly) that it was Notepad that edited the filw web.config.  But even with this info, we cannot tell what was actually changed within the file!  (Was it a harmless addition of an application extension? Or did the user modify an important entry within the file?)

To know what was changed, we would now have to access a file server backup, and perform a file compare on the old and new versions. Doable, but that’s a heavy burden to answer a pretty straightforward question: “What did the user change???”

Event Viewer: But what was changed?
What User Activity Monitoring shows you: ObserveIT’s log shows what the user did, in a concise and descriptive manner. And again, video replay shows what took place within the file.


Video replay of editing the web.config file.

Scenario 4: Changing port used by IIS

What the User Did: An admin user changed IIS to listen to port 8080 instead of the default 80. This was done via “Start > Settings > Control Panel > Administrative Tools > IIS Manager”, and once there editing the Properties for “Default web site”.

Set IIS to listen to port 8080
Security and Audit Implications of this Action: Modifying the port of a service accessible from outside the DMZ can open a huge hole in the firewall security.
What shows up in system event logs: Among the 5,500 log entries, there is one entry that adds IIS Manager to the Recent Items list in Windows. This is timestamped when the app was closed, which might mislead the investigator, and also wouldn’t even occur if the user left the window open. Earlier, there is an obscure log entry indicating a DLL that was loaded to memory. This is the true indication that IIS Manager was launched, but it is very difficult to find this in a reasonable level of effort!

Event Viewer: Obscure log entry of DLL.
It turns out that this is the culprit!
What User Activity Monitoring shows you: Once again, ObserveIT gives us the whole picture.


Video replay of changing the IIS port.

Platform Considerations

The above experiments were performed on a Windows Server 2003 . Windows Server 2008 R2 has added additional audit policy granularity. However, these updates do not mean that additional knowledge can be gleaned from the logs. Only that the logs can be filtered a bit better. The bottom line remains that many high-risk, security-impacting actions, including those highlighted in this paper, are not logged.

Conclusion

Security audits that rely on existing system logs have large holes in them due to the fact that system logs simply do not capture the relevant information necessary.
For issues that are known a priority, the blind spot can be eliminated with a custom utility targeted at that specific issue. But this only solves this one specific issue.
The easiest way to fill these blind spots in their entirety is by adding User Activity Monitoring such as ObserveIT, which augments the existing system and database logs by showing precisely what the user did (as opposed to the technical results of what he did.)

About ObserveIT

ObserveIT User Activity Monitoring software meets the complex compliance and security challenges related to user activity auditing, one of the key issues that IT, Security and Compliance officers are facing today.
ObserveIT acts like a security camera on your servers, generating audit logs and video recording of every action the user performs. ObserveIT captures all activity, even for applications that do not produce their own internal logs. Every action performed by remote vendors, developers, sysadmins and business users is tied to a video recording, providing bulletproof forensic evidence.
ObserveIT is the ideal solution for 3rd Party Vendor Monitoring, and PCI/HIPAA/SOX/ISO Compliance Accountability.
Founded in 2006, ObserveIT has a worldwide customer base of Global 2000 companies that spans many industry segments including finance, healthcare, manufacturing, telecom, government and IT services.