In this review, we compared three log managers: VMware’s Log Insight, Balabit’s Syslog-ng Professional Edition, and SpectorSoft’s SpectorSoft Server Manager. Each offers a way of gathering, compiling, and in the case of VMware, and to a lesser extent, SpectorSoft, making sense of syslogs and Windows Events.
Each vendor’s approach has strengths and decided weaknesses. For syslog and messaging tracking, Syslog-ng Pro is tough to beat as it digests almost anything, works on a vast number of platforms, and has highly tunable message filters. It does not, however, do any analysis—although it will happily cram popular database packages to the gills, at high speed, with filtered, time-stamped log messages.
VMware’s Log Insight can be an almost-must have for VMware infrastructure. It handles a wide variety of log sources via host-installed agents, and has free agent add-ins that add specific brand/model/OS/app details. What’s missing: a larger number of partner/product-specific plug-ins, at least for now. The upshot is that its analysis and dashboard representation of the analysis is very strong, if not quite as vastly heterogeneous as Balabit’s syslog-ng Pro’s.
+ ALSO ON NETWORK WORLD: Portnox, Extreme lead NAC pack +
SpectorSoft Server Manager is hosted on Windows, but Linux or any other server capable of transmitting syslog information can play just as easily. It’s a good candidate for SMBs and branch offices, and its GUI and feel will be familiar to Windows admins, and without the overhead of Microsoft’s gargantuan Systems Center.
Logging basics
There are three accepted file formats found via RFCs, and beyond formatting and denoting various fields, there are no real rules. Each app and operating system handles messaging more culturally than by rulebook. Windows does events logging differently still, and in graduated formats depending on the age of the Windows OS version. Older Windows apps write their own logs, which by convention, are written in a temp directory, or in reality, anywhere an app vendor chose. In our testing, we were interested in the common sources, /var/log messages, and Windows Events.
In Unix variants, logs can be written to syslog files in one of two syslog formats, or they can be written to the traditional /var/logs directory as standalone files in formats that don’t follow any particular standard. Most apps obey the rules, indicating app sources (facility type, a number between 0 and 23), severity (criticality, a number between 0 and 7), and a formatted message that can be parsed from message to message with continuity and framing stamps.
Many apps can mean many messages, and operating systems and their log entry apps enforce varying rules on when messages are compressed, or how large they can grow. Busy and verbose systems infrastructure can generate 100GB of traffic per day, and so log compression and rotation are necessary in log servers and their analyzers—yet audit and regulatory compliance may mandate storing log entries in toto (if compressed), for a very, very long time.
The meaning of these messages can range from informational to one of several states where the aggregation of seemingly informational messages establishes a trend to worry about. Or there might be a priority message that has important meaning requiring action or forensic behavior, or both.
Since hosts can have different operating systems and software versions, log managers must be able to digest from all critical sources. A special emphasis on heterogeneity is important, because the concept and fallacy of secure perimeters is a hacker’s delight, and therefore most all hosts in a network operation center need to have their logs examined.
Here are the individual reviews:
VMware vRealize Log Insight 2.5
VMware’s re-branded vRealize Operations Manager integrates VMware ESXi 5X+ infrastructure (5.0, 5.1, and 5.5) along with server syslogs gathered the traditional Unix log server way, along with data gathered from agents.
It then presents storage and most importantly, trend analysis of these logs. While not vastly heterogeneous, an increasing number of agents, along with good trend and alarm analysis, make Log Insight very useful for system administrators, support staff, and network engineers.
Once configured, and our experience is that no log server or manager is easy to configure, Log Insight 2.5 can offer highly detailed trend analysis in its dashboard with customizable graphs, or as a systems trends and alarms manager.
During sustained traffic, the UI of Log Insight can be slowed; if an anvil drops on its foot, it might be up to 30 seconds before it screams ouch, but it screamed reliably when we dropped anvils in testing. We noted that Log Insight 2.5 has no way to react to Simple Network Management Protocol (SNMP) traps of any kind.
Log Insight accepts logs in three ways. These include native log collection from ESX infrastructure, syslogs sent from other hosts at the host’s operating system level via customized VMware Marketplace (free) agents from Cisco, Brocade, and/or custom made agents for items like Microsoft SQL Server, Active Directory, and general Windows Events. The agents run as system processes/daemons.
These data sources push their logs to Log Insight Manager at intervals, not necessarily giving immediate reactions, like SNMP does, but in a way that can be made more secure than SNMP, as SNMP requires a lot of work to ensure secure communications with an SNMP responding app or manager.
Log Insight 2.5 is supplied as a virtual machine (VM) appliance, one version for VMware, and another for Microsoft Hyper-V, both built on SUSE Linux. All log entries are amalgamated into a quickly growing pile of messages that Log Insight can filter, digest, normalize, and document.
Log Insight needs lots of storage and as a VM, requires lots of host horsepower (CPU+memory+storage) to keep up, as did the other products we compared. Logs can be compressed, then discarded by admin settings. There are four appliance settings ranging from Extra Small, Small, Medium, and Large, each with ingestion limits imposed, and graduated CPU, memory, and storage capacities set at installation. We chose Small in our tests.
Setup requires a stable DHCP infrastructure, as moving the VM into a different IP addressing scheme is a major hassle. We used the OVA (pre-packaged VM with metadata) version, and a normalized VMware ESXi infrastructure.
Logs are communicated for VMware hosts simply, and Log Insight sends warning emails of absent communications among VMware hosts (but not non-agent syslog sources) if communications are lost. Installed agents are placed into Windows and Linux hosts, and the messages are communicated to the Log Insight VM instance, where they’re digested, processed, and presented in a supplied or administrator-designed web page dashboard.
Log Insight sets itself above others when its dashboard reacts to the changing circumstances of log reports sent to it. This is only mildly offset by the fact that it’s not instantaneously responsive, and we found that reaction times can be up to 70 seconds — even on state-of-the-art hardware. The lack of responsiveness doesn’t seem to be a function of growing log database size, and we offer only conjecture here, that imposing sophisticated rules just takes time for digestion. Log Insight can be crippled in responsiveness by limiting its desired memory size or CPU allocation—it eats both of these, thoroughly.
Like the other products we tested, it’s possible to use relays and high availability can be built-into the VMware design, so that critical logs at peak times can continue to flow. How that’s done is by placing servers in strategic spots so that an outage doesn’t incapacitate error logs.
The native high availability capabilities of VMware’s methods, however, allow servers to keep on working as Log Insight will forcibly slow its input during peak periods. Agents used by Log Insight have the ability to throttle traffic flow during congestion periods, which ostensibly back-off servers from sending messages until congestion can be removed, however, we could not throw enough messages or create sufficient events to be able to verify if this is being done.
Absent entries are also the crux of a debate as regards whether to use the UDP protocol, TCP, or something secure between a syslog/event-tracking server and its constituent log-sending hosts. Older syslog protocols used UDP, which can drop messages during congestion, or in other ways (man-in-the-middle attacks, misconfiguration, ARP poisoning) simply not get the message there. TCP can now be used, but congestion problems remain.
What also remains is the theory of the plain-text components of syslog messages that allow simple protocol analyzers to read traffic, and obtain privileged information as a result of this. This issue is sometimes made further complex by organizations that create management backbones for management and control backplane traffic, such as that carried for SNMP, iLO, and other types of management/admin/control traffic.
Surreptitious interception of the traffic gives knowledge of hosts and their configuration for cracking or tracking purposes. Towards these ends, Log Insight uses TLS over TCP protocols from agented hosts by default, although it can accept unencrypted syslog message traffic over UDP (not recommended) or TCP (better, but still not as desirable as encrypted traffic).
What’s missing
No SNMP monitoring is available unless it’s somehow client- or app-triggered into syslog traffic. Missing hosts can’t be found until a log collection interval has passed; there is no ping or analogous function we could find to regularly see if hosts are alive. We could find amazing details about storage and logon detail, and both trivia and critical alarms, but the primary health indicator of: host is alive, isn’t there.
The VMware vRealize Operations Manager (aka “vcops”) adds more operational functionality that isn’t the crux of this review, but does make up for shortcomings in Log Insight. It costs extra.
The Log Insight 2.5 package comprises log compilation and very good analysis for the devices it covers. That list includes VMware (of course), Windows, Linux, Cisco and Palo Alto products, popular database applications, and this list grows. Agents are free, and installable by a menu pick. It works well for the products covered, and gives great promise to Situation Room analysts of many types. It’s configurable, redundant, and firmly VMware-like.
Balabit Syslog-ng Pro Edition 5.0 Long Term Support (Three Years)
As a log consolidation engine, open source syslog-ng has a long and positive reputation in the Linux, AIX, and BSD communities. The rsyslog package has recently replaced syslog-ng on some notable host Linux distributions, such as Red Hat/CentOS and Ubuntu, but both open source syslog-ng and Syslog-ng Pro support these, and many others.
Shifts away from including syslog-ng in distributions aside, the Syslog-ng Pro version has astoundingly heterogeneous support for varying operating systems. Balabit differentiates the open source version from the Pro edition by including support, and Windows-specific hosting and client-side communications for log consolidation from Unix-ish and Windows hosts alike. The “-ng” stands for “next generation”.
Syslog-ng Pro takes one of three roles in a host, as log server, a relay to a log server, or as a monitored host. Messages generated by the monitored host OS or applications in the host are initially parsed by a syslog-ng (Pro) agent to be sent to a destination (or be ignored).
Surviving messages are then made into a log statement. The messages are shipped across the network to one or more destinations, subject to rules in the sending host. Messages communicated to the log server can be parsed by the log server host, subject to filtration, rewritten into a different format, or even elevated as to criticality depending on simple filters. These entries are in turn, sent to final destinations, which include varying files/database tables.
The log server is the central repository of the logs, and the logs can have much filtration before they arrive in one of several databases supported by syslog-ng Pro; we tried MicrosoftSQL, and MySQL. Output destinations formed from JSON expressions, as well as direct calls to MongoDB, are possible, too.
From records either in the flat files formed of message records, or as the result of external apps like Splunk or queries made to a database, trend analysis can be formed, and/or simple scripts can be built to act on messages within the datastore.
Balabit’s Syslog-ng Pro doesn’t analyze logs; an external app must do this. Syslog-ng and Pro can consolidate, normalize and then store the data in the logs, always stamping them with time and source. Data subsequently stored in the log server is usually encrypted, but there are ways to store the data as plain text, if desired, and we hope you don’t.
The use of relays is recommended, as the amount of traffic that can be generated on even our small test network can become huge.
It’s possible to use encrypted and modulated traffic protocols for Windows hosts, the RLTP protocol, which can modulate traffic under congestion conditions. TLS-TCP, which is an encrypted protocol, can also be used as an encrypted networking protocol between hosts, but it isn’t modulated in the same way that the RLTP protocol controls flow.
On the designated log server, it’s possible to have trigger events, which can include talking to an SMTP/mail server to send an alert. A cumulative number of messages that add to a trend, however, can’t be acted upon internally by Syslog-ng Pro.
Downsides
While highly configurable, it’s not plug and play. Gaining high efficiency requires tuning the messages, and knowing the nature of them in apps and the hosts they play on. Basic productivity can come quickly, but there is much surgery that can be done to limit both flows, and increase response time.
The upside to this is that the flows and options are very well documented, and although poised towards systems engineers, the docs were very good. Rarely do we praise documentation, but we found the docs to be understandable, and had plenty of useful examples—and there’s almost 500 pages of them.
If you do one thing, do it very very well, and Syslog-ng Pro is highly tunable, and fast. Hosts rarely had their Syslog-NG Pro processes rise above 6% CPU. We warn that log servers must, like VMware’s Log Insight hosts, be highly provisioned and healthy machines. Larger organizations may take umbrage with the fact that most all hosts need to be licensed, but from a regulatory and compliance perspective, it’s a small tax to pay.
SpectorSoft Server Manager
This application felt more to us like a traditional server management system in that it accepts logs from Windows, Linux (or other syslog hosts), as well as SNMP messages, merging them in a console app that can also publish systems health status reports to web pages.
+ ALSO: SpectorSoft touts less threatening employee monitoring software +
It doesn’t have the vast heterogeneous log sources as VMware Log Insight or Balabits Syslog-ng Pro, but for many in especially smaller, but busy networks, its look and feel will be familiar.
The Server Manager application lives on a Windows host, and preferably in an Active Directory network. It will accept log sources and syslog data from Linux, and we were also able to send it Apple OSX logs, just to see if we could cause any explosions. There were none.
This is an action program. One “fires an alert”, rather than merely “spawns an alert”. For this reason it appealed to our admittedly over-testosteroned admin instincts.
Server Manager is very Active Directory-focused, and can climb trusted or untrusted forests easily, if administrative credentials are handy. Lists of hosts can also be imported, along with credentials for Windows Management Interface, if handy. This doesn’t limit what can be monitored or logged, it’s just that much of the product is about Windows Management and triggers based on Windows logs and the events listed in the logs.
Other hosts, be they Linux, BSD, or devices and appliances that support Secure FTP (SFTP), and/or SNMP can also be monitored/polled for information.
Templates are available that are administratively defined objects surrounding common WMI-monitored Windows characteristics, or syslog data, or SNMP traps. We had to have a working POP email address (Exchange will do, too, if it will accept Outlook-like credentials) for fired off alerts, based on conditions we set within the objects. Monitors are used to check criteria at an administratively defined frequency, and their history can be retained for a while, but even this history over many computers becomes large, and retention for a few days is likely to be recommended.
For Active Directory monitored hosts, there is an Auto Configurator, which climbs through the directory tree, using pre-determined templates to “automagically” turn on monitors for new hosts, or added branches of the directory tree via administrative logons, which then chain the designated hosts as entries into the pool of hosts.
There is a temptation to be overactive in monitoring, because it’s easy to do. This service isn’t available for Linux or MacOS hosts.
Some templates are the crux of an additional product from SpectorSoft that monitors disk conditions, but is optional, and was not tested. Host groups can be organized by varietal type, or other purposes using templates and monitors. This works well in differing versions of Windows, where the OS version or app payload dictates monitoring differing features. Perhaps the accounting department has desktops running SQL Server, or a SharePoint farm uses IIS, each with perhaps differing monitoring needs/requirements.
Reports can be viewed, or published periodically to a web page, although the details are left to the administrator. Many are focused on Microsoft Event and Security logs (failed logons, lockouts, etc) but we could also watch the syslogs for specific events.
Making syslog-event reports required quite a bit of work because of the extremely varied messages possible, but it’s also possible to use keywords and priority data to summarize them. Reports are published to individual pages if desired, and cannot be listed in a control-panel like amalgamation of cross-platform events easily (requires some coding).
The canned Windows reports were nicely verbose, and allowed us to see items like lockouts (and the associated code as to why), new accounts, object access, and registry status. We could include or exclude various times or days, and get good summaries. Filters are also available to use on either consolidated log entries, realtime data not archived, and use these filters on monitors or reports.
Actions can be taken based upon object monitors that we could set, but these are useful almost exclusively on Windows events. The actions in Windows go deeper than many management apps we’ve seen. Monitored objects can spawn alerts, database entries, or can start/stop Windows system processes. For other operating systems hosts monitored, all that can be done is to insert a line into the host’s syslog file.
Controlling noise through filters is important, so that bogus information doesn’t suddenly spawn several hundred emails, or desktop messages to administrators. You can target control groups with messages, and emails and alerts to your heart’s content.
Log files can be consolidated, encrypted, digitally signed, and centrally backed up, as is the case with the other two products we tested, so as a log server, it satisfies this important requirement.
Downsides
The SpectorSoft Server Manager is highly focused towards Windows, and for heavy Windows installations, this is a good thing, and for vastly heterogeneous infrastructure, it means more work is needed. There is much depth with this product, and it begs to have the optional disk-focused product licensed. Account information always has a test button to verify that hosts can communicate, mail can work, etc. We like this step. We did encounter a few headscratcher type errors that went away when we logged off the Server Manager host, and logged back on again, but no data or work was lost on these occasions.
The SpectorSoft Server Manager can be built quickly, once its docs and processes are understood. Windows is the primary management/log server target of Server Manager, but yes, it’ll work with those pesky Linux instances you may have. Once similar depth with non-Windows hosts occurs, Server Manager could become a larger organization’s primary health tool.
Summary
At the end of using these three, we wish we could mosh them together somehow, as they each have strengths. VMware’s Log Insight has a great set of customizable dashboards, and decent/burgeoning heterogeneity of log types, all nicely and securely communicating through free-as-in-beer, well-defined host agents.
Balabit has mastered the heterogeneity and filtration mechanisms, and seems ready for large organizations, which likely are already familiar with syslogd and syslog-ng. SpectroSoft’s depth into Windows infrastructure as a management app, rather than an analysis tool, would be a delight if we could couple it with the other two.
Do you need all three? No. Microsoft would want us to argue that its app is the most appropriate for Windows, but it does little else in host log management. Red Hat would have their say. The three products take differing approaches to how to do log servers, and in the wake of recent high-profile enterprise breaches, a variety of tools is a good thing.
How we tested log managers
We used log sources from OSX 10.6/9/10, CentOS 7, Red Hat 7, Ubuntu 14.10, Windows Server 2008R2, Windows 2012R2, Windows 7 and 8.1 hosts, some as virtual machines (Windows Hyper-V3, VMware 5.1/5.5, Parallels for Mac) and some as native OS against installations of each of the three products.
We hosted Hyper-V with Windows and Ubuntu instances, and VMware with 5.1 and 5.5 hosts. We used a Lenovo RD630 server (16GB/4TB disk) for bare metal tests, and hosted hypervisors on HP DL560Gen8 (32proc/64GB/2TB disk), HP DL580Gen8 blades (60proc/256GB/12TB disk).
VMware vRealize Log Insight 2.5 was supplied as a loadable appliance (OVA) and we tested it primarily on VMware ESXi 5.5. Balabit Syslog-ng 5.0 was tested on Red Hat 7 bare metal, CentOS7 as a VMware ESXi 5.5 VM, and on Windows 2008R2. SpectroSoft Server Manager was tested on Windows 7 on a VMware ESXi 5.5 host initially, then changed to a Windows 2008 R2 Standard, connected to a dual-trusted forest Active Directory infrastructure hosted on Hyper-V and ESXi 5.1.
We had fun failing logins, filling disks to alarming levels, and causing general mayhem in an attempt to fill event and syslogs quickly, taking hosts offline and allowing them to accumulate large log payloads, then turning their ports back on again to see how products reacted, which in most all cases, showed they weren’t fazed much by traffic on our switched 10Gigabit Ethernet/1G Ethernet network (Extreme Summit Switches, port controlled). We used all products via a remote VPN connection to our hosted NOC at Expedient, an ISP/MSP in Carmel Ind.
Tom Henderson runs ExtremeLabs, in Bloomington, Ind. He can be reached at kitchen-sink@extremelabs.com.
