Anyone who is concerned about their Linux servers' security, stability, and proper functioning needs to audit their systems. Auditing may include anything from logging simple Bash commands to following complex system processes. Linux auditing can be a complex process, but here are some basic tools and techniques you can use to make the task simpler.
First, let's talk about some simple ways you can get some idea of what users do that don't reach the level of auditing. You can, for instance, check the shell command history using Bash's built-in command
history. It shows the last commands executed by the current user. To see the date and time of executions, reconfigure history's settings with the command
export HISTTIMEFORMAT='%F %T '.
history command acts only upon the current user. To see other users' activity, provided you have permission, read the hidden file called .bash_history in their home directories. One caution, however: The Bash history can incomplete, making the record meaningless. If you see in it
./myscript.sh, all you know is that the user executed a script; you can't tell what was in this script or what it did, unless of course the script remains available and unchanged.
To see is who is currently logged in and what each user is doing, use the command
/usr/bin/w, which gives you this information in full.
Many services, including a couple of popular databases, provide logs for simple auditing. For MySQL a hidden file called .mysql_history in users' home directories logs all the user actions in the MySQL console. A similar file for PostgreSQL is called .psql_history.
History logs frequently contain passwords or other sensitive data. Consider, for example, the commands they might show when you create MySQL or PostgreSQL users. Therefore, make sure that history log files can be read (and of course written) only by each user; they should have chmod permission flags set to 600. If you are more concerned about security than auditing, you can delete these history files entirely, then create a soft link to /dev/null in their place. For example, for MySQL, run the command
ln -s /dev/null ~/.mysql_history.
These basic practices are just a start. They touch only the surface of what users do and what happens on the system, and it's trivial for an attacker to delete logs showing his traces. You therefore need an advanced auditing solution to not only reveal malicious activity but also store this information in a secure, preferably remote place.
Advanced Linux auditing
The Linux Auditing System is a Linux kernel implementation available in CentOS and other distributions that enables in-depth and advanced auditing. It works on the kernel level, where it can oversee every process and activity on the system. It uses the auditd daemon to log what it finds.
In most Linux distributions auditd is preinstalled and starts and stops automatically with the system. It logs information according to its auditing rules, and also conveys SELinux messages as described in our article about Linux server hardening.
Auditd's configuration is controlled by a few files in different directories. The daemon's configuration file is /etc/audit/auditd.conf, which contains all the settings except the auditing rules. Leave the default values in place while you're still exploring auditing. Important settings include
max_log_file (default 6), the maximum size of the log file in megabytes. Once a log file reaches this limit, the action specified in the setting
max_log_file_action (default rotate) takes place. The setting
num_logs (default 5) specifies the number of log files and thus determines how many log files are kept.
The file /etc/audit/audit.rules contains the auditing rules that control what events should be audited and logged. You can specify three types of options in this file: control, file system, and system call.
The control options manage the system rather than the auditing rules. For example, the audit.rules file should always start with a directive that any existing auditd rules are deleted (
-D). Another useful control option is
-e 2, which makes the configuration immutable and requires a server restart for new changes to take effect.
File system rules pertain to files and directories recursively, and each rule looks like this:
-w <em>path-to-file</em> -p <em>permission</em> -k <em>keyword</em>
All file system rules begin with
-w, which stands for watch. A permission is an action that reads (
r), writes (
w), executes (
x), and/or changes the attribute (
a) of a file. The keyword is an intuitive you choose to connect to one or more auditd rules. The same keyword can be used for more than one rule.
An example should help illustrate how file system rules work. The rule below instructs auditd to watch the file /etc/shadow, the Linux password file, for being read, written to or having its attributes modified:
-w /etc/shadow -p rwa -k shadow_watch
When a rule is tripped, auditd writes a log entry to its log file /var/log/audit/audit.log.
If you make any changes to the audit.rules file you must restart (or reload) the auditd daemon with the command
service auditd restart. If you enter the rule above, you can test it by restarting the daemon, then trying to read /etc/shadow. Next, search the current autitd log using the
ausearch command with the keyword for the rule in question:
ausearch -k shadow_watch -i. The result should be similar to:
type=PATH msg=audit(11/18/2012 16:24:19.963:61) : item=0 name=http://www.openlogic.com/etc/shadow inode=163882 dev=fd:00 mode=file,000 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:shadow_t:s0 type=CWD msg=audit(11/18/2012 16:24:19.963:61) : cwd=http://www.openlogic.com/root type=SYSCALL msg=audit(11/18/2012 16:24:19.963:61) : arch=i386 syscall=open success=no exit=-13(Permission denied) a0=bfde58e9 a1=8000 a2=0 a3=1 items=1 ppid=2148 pid=2149 auid=root uid=anatoli gid=anatoli euid=anatoli suid=anatoli fsuid=anatoli egid=anatoli sgid=anatoli fsgid=anatoli tty=pts0 ses=1 comm=cat exe=http://www.openlogic.com/bin/cat subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=shadow_watch
The ausearch argument
-i tells the command to interpret numbers; that is, uid 503, for instance, is translated to "anatoli." The above log excerpt shows that the user anatoli has tried to read the file /etc/shadow (name) using the executable
/bin/cat (exe). The log shows
success=no, the attempt was unsuccessful, and the exit code was -13, which means permission denied.
ausearch utility lets you filter your results. To see all unsuccessful attempts, use
-sv no, where sv stands for success value. The full command with the shadow keyword would be
ausearch -k shadow_watch -sv no. For more information about the returned values and ausearch, check its manual page by running the command
The third type of option, system call (syscall), provides the interface between an application and the Linux kernel. These auditd rules act upon the specified interfaces to detect and log events. System call auditd rules have the following structure:
-a <em>when</em>,<em>filter</em> -S <em>system-call</em> -F field=<em>value</em> -k <em>keyword</em>
-a stands for append – that is, append the rule at the end of the ruleset. You can also use
-A to place it at top of the list, or
-d to delete the rule.
when values are
never, meaning always or never to create an event log.
For the third argument,
filter, two values are frequently used:
exit value means to act upon a syscall exit, when an operation is completed. The
user filter is for userspace events and can be further filtered to
The next argument is
-S followed by a syscall name. There are hundreds of syscalls – to see all of them, go to the syscalls man page – and often more than one can be used to get a similar result. For example, if you want to see whether a file or directory has been deleted, you could use
The last argument before
-k for keyword is
-F, which stands for fine-tune filter field. If you go back to the
ausearch -k shadow_watch result, you can see the numerous fields that can be used for fine-tuning the rules and hence the results. A sample fine-tuned rule looks like this:
-a always,exit -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k delete
This rule covers syscalls that may lead to a file disappearing. The two fine-tuning rules (
-F) state that in order the rule to be tripped the user's ID should be above 500 (regular users) but should be different from 4294967295, which is the representation of the auditd system ID. For further information about the fine-tuning fields, check the
audit.rules man page.
To get you started faster with the auditd rules, CentOS 6 provides some example rules in the file /usr/share/doc/audit-2.2/stig.rules. Just using them without any adjustment provides a solid ground for auditing.
Auditd log interpretation
Once you've set the events you want to track, you can do the actual auditing by interpreting auditd's log file at /var/log/audit/audit.log. The file contains the same information given by
ausearch but in more user-unfriendly format. One problem that you may encounter, if you decide to read the log file directly, is that the time is given in Unix timestamp format, which means you have to convert the timestamps to readable dates and times in order to tell when an event occurred.
Many tools can make it easier to read and analyze information from auditd's log. The aureport utility, for instance, lets you generate reports from the auditd log file. Running just
aureport provides you with an easy-to-understand summary report. It includes counters for all important auditing event groups, such as number of changes to accounts, groups, or role. Detailed reports are also available, and they can be filtered by type (file system or syscalls), fields, and time. Here are a few useful
--auth– shows authorization attempts. The aureport command can be further extended with the
--failedargument to show failed attempts only and with
--startto limit the time frame of the report. Thus to see failed logins for yesterday use the command
aureport --auth --failed --start yesterday
--key– lists events for keywords defined in the auditd rules
--file– shows events for specified files and directories
--syscall– reports on system call events that have been configured to be logged in the auditd rules
For more details about these reports and to find out more information about aureport check its manual page (
Auditd reporting has to provide genuine and reliable information. To make this possible the auditing cycle has to be secured and hardened. Auditing hardening enforces best practices for ensuring reliability, integrity, and security of the auditing process.
The first step for hardening is to ensure auditd's configuration is immutable by using the control option
-e 2. Next, ensure the logs are stored in a secure centralized location. The best place is a server dedicated to accepting remote syslog events.
A utility called audispd (auditd's dispatcher) can help with this task, along with one of its plugins, audisp-remote. Audisp-remote allows events to be sent to a remote syslog server. Its configuration can be found in the /etc/audisp/audisp-remote.conf file. Here's a sample configuration specifying that a remote rsyslog server 10.0.0.1 is listening on port TCP 514:
remote_server = 10.0.0.1 port = 514 transport = tcp
Having events logged and stored safely to a remote location helps provide peace of mind. Of course there's no guarantee that the remote server cannot be corrupted or compromised, but doing this adds one step more toward better system administration and security. That's why reliable remote logging is a requirement for financial and government environments.
Linux auditing can be as simple as reading simple history log files, or it can be a real challenge if you decide to be serious about it. For reliable auditing, use the powerful Linux kernel options and the supplementary services auditd and audispd.