Making better use of your Linux logs

Log files on Linux can provide a lot of useful information on what's happening on your system. The below commands can help you sort through the data and pinpoint problems.

Making better use of your Linux logs
Nishan Joomun (CC0)

Linux systems maintain quite a collection of log files, many of which you are probably rarely tempted to view. Some of these log files are quite valuable, though, and options for exploring them might be more interesting and varied than you imagine. Let's look at some system logs and get a handle on some of the ways in which log data might be easier to probe.

Log file rotation

First, there's the issue of log rotation. Some Linux log files are “rotated.” In other words, the system stores more than one "generation" of these files, mostly to keep them from using too much disk space. The older logs are then compressed but left available for a while. Eventually, the oldest in a series of rotated log files will be automatically deleted in the log rotation process, but you’ll still have access to a number of the older logs so that you can examine log entries that were added in the last few days or weeks when and if you need to look a little further back into some issue you're tracking.

To get a feel for what types of system information are being saved, simply cd over to the /var/log directory and list its contents.

/var/log# ls
alternatives.log btmp.1 kern.log.2.gz syslog.3.gz
alternatives.log.1 cups kern.log.3.gz syslog.4.gz
alternatives.log.2.gz dist-upgrade kern.log.4.gz syslog.5.gz
alternatives.log.3.gz dpkg.log lastlog syslog.6.gz
alternatives.log.4.gz dpkg.log.1 mail.err syslog.7.gz
alternatives.log.5.gz dpkg.log.2.gz mail.err.1 sysstat
apport.log dpkg.log.3.gz mail.err.2.gz tallylog
apport.log.1 dpkg.log.4.gz mail.err.3.gz ufw.log
apt dpkg.log.5.gz mail.err.4.gz ufw.log.1
atop faillog mail.log ufw.log.2.gz
auth.log fontconfig.log mail.log.1 ufw.log.3.gz
auth.log.1 gdm3 mail.log.2.gz ufw.log.4.gz
auth.log.2.gz gpu-manager.log mail.log.3.gz unattended-upgrades
auth.log.3.gz hp mail.log.4.gz wtmp
auth.log.4.gz installer speech-dispatcher wtmp.1
boot.log journal syslog
bootstrap.log kern.log syslog.1
btmp kern.log.1 syslog.2.gz

This is fairly large collection of logs and log directories — 69 files and directories in /var/log in this case, but 180 files when you include the files inside those directories.

$ cd /var/log
$ ls | wc -l
69
$ find . -type f -print | wc -l
180

When you examine your log files, you will see pretty clearly which are generations of the same basic log. For example, one of the primary log files — the syslog file — is broken into nine separate files. These represent what is basically a week's worth of historical data along with the current file. Most of the older files are zipped to preserve space.

$ ls -l syslog*
-rw-r----- 1 syslog adm 588728 Oct 15 20:42 syslog
-rw-r----- 1 syslog adm 511814 Oct 15 00:09 syslog.1
-rw-r----- 1 syslog adm  31205 Oct 14 00:06 syslog.2.gz
-rw-r----- 1 syslog adm  34797 Oct 13 00:06 syslog.3.gz
-rw-r----- 1 syslog adm  61107 Oct 12 00:08 syslog.4.gz
-rw-r----- 1 syslog adm  31682 Oct 11 00:06 syslog.5.gz
-rw-r----- 1 syslog adm  32004 Oct 10 00:07 syslog.6.gz
-rw-r----- 1 syslog adm  32309 Oct  9 00:05 syslog.7.gz

The syslog files contain messages from many different system services — cron, sendmail and the kernel itself are just examples. You'll also see evidence of user sessions and cron (scheduled tasks).

Most Linux systems no longer use the old messages and dmesg files that served as landing places for the bulk of our system messages for many years. Instead, a large variety of files and some special commands have become available to help present the log information that is likely to be most relevant to what you are looking for.

Depending on the file in question, you might simply use more or tail commands, or you might use a file-specific command like this use of the who command to pull user login data from the wtmp log.

$ who wtmp
shs      pts/1        2018-10-05 08:42 (192.168.0.10)
shs      pts/1        2018-10-08 09:41 (192.168.0.10)
shs      pts/1        2018-10-11 14:00 (192.168.0.10)
shs      :0           2018-10-14 19:11 (:0)
shs      pts/0        2018-10-14 19:16 (192.168.0.25)
shs      pts/0        2018-10-15 07:39 (192.168.0.25)
shs      :0           2018-10-15 19:58 (:0)
dory     pts/0        2018-10-15 20:01 (192.168.0.11)
shs      pts/0        2018-10-15 20:42 (192.168.0.6)
shs      pts/0        2018-10-16 07:18 (192.168.0.6)
nemo     pts/1        2018-10-16 07:46 (192.168.0.14)

Similarly, you might see nothing when you run a tail faillog command, but a command like this shows you that it's simply full of zeroes:

# od -bc faillog
0000000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000
         \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
0076600

You may also see very little when you try to tail lastlog only to discover that you need to use the lastlog command to view that log's data.

So, here is a listing of log files in /var/log and some descriptions of what they contain and how to view their contents.

  • alternatives.log -- "run with" suggestions from update-alternatives
  • apport.log -- information on intercepted crashes
  • auth.log -- user logins and authentication mechanisms used
  • boot.log -- boot time messages
  • btmp -- failed login attempts
  • dpkg.log -- information on when packages were installed or removed
  • lastlog -- recent logins (use the lastlog command to view
  • faillog -- information on failed login attempts -- all zeroes if none have transpired (use faillog command to view)
  • kern.log -- kernel log messages
  • mail.err -- information on errors detected by the mail server
  • mail.log -- information from mail server
  • syslog -- system services log
  • ufw -- firewall log
  • wtmp -- login records

journalctl

In addition to the log files maintained in /var/log, there is also the systemd journal. While not a simple "log file" in the usual sense of a single log file, this journal represents an important collection of information on user and kernel activity. The information is retrieved from a variety of sources on the system.

To view the information that has been collected, you would use the journalctl command.

How much information you will see depends on whether you are a member of the adm group or not. Non-adm users will see relatively little information, but members of the adm group will have access to a massive amount of data — as shown in this example, which is merely showing us how many lines of information are available for this adm group member to review:

$ journalctl | wc -l
666501

That's more than 666,000 lines of text! To pare this down to a hopefully more digestible display, you're probably going to want to use arguments that tailor what you will see displayed. Some of the options available with journalctl include:

--utc (change the time format to UTC)
-b (only show records added since the last boot)
-b -1 (only show records added since the previous to last boot)
--since and --until (only show records added within the specify timeframe, e.g., --since "2018-10-15" --until "2018-10-11 06:00"

Here's an example:

$ journalctl --since "2018-10-16 13:28"
-- Logs begin at Mon 2018-05-14 15:16:11 EDT, end at Tue 2018-10-16 13:28:57 EDT. --
Oct 16 13:28:25 butterfly kernel: [UFW BLOCK] IN=enp0s25 OUT= MAC=01:00:5e:00:00:01:02:
Oct 16 13:28:25 butterfly kernel: [UFW BLOCK] IN=enp0s25 OUT= MAC=01:00:5e:00:00:fb:00:
Oct 16 13:28:57 butterfly su[7784]: pam_unix(su:session): session closed for user root
Oct 16 13:28:57 butterfly sudo[7783]: pam_unix(sudo:session): session closed for user root
lines 1-5/5 (END)

You can also examine log entries just for some particular service. This is probably one of the more useful things that the journalctl command can do for you:

$ journalctl -u networking.service
-- Logs begin at Mon 2018-05-14 15:16:11 EDT, end at Tue 2018-10-16 08:06:31 EDT
May 14 15:16:12 shs-Inspiron-530s systemd[1]: Starting Raise network interfaces.
May 14 15:16:12 shs-Inspiron-530s systemd[1]: Started Raise network interfaces.
May 14 15:49:18 butterfly systemd[1]: Stopping Raise network interfaces...
May 14 15:49:18 butterfly systemd[1]: Stopped Raise network interfaces.
-- Reboot --
May 14 15:49:50 butterfly systemd[1]: Starting Raise network interfaces...
May 14 15:49:51 butterfly systemd[1]: Started Raise network interfaces.
-- Reboot --

Notice how the system reboots are displayed in this output.

To get a list of services, try a command such as this:

$ service --status-all | column
 [ + ]  acpid                    [ + ]  network-manager
 [ - ]  alsa-utils               [ - ]  networking
 [ - ]  anacron                  [ - ]  plymouth
 [ + ]  apparmor                 [ - ]  plymouth-log
 [ + ]  apport                   [ - ]  pppd-dns
 [ + ]  atd                      [ + ]  procps
 [ + ]  atop                     [ - ]  quota
 [ + ]  atopacct                 [ - ]  quotarpc
 [ + ]  avahi-daemon             [ - ]  rsync
 [ - ]  bluetooth                [ + ]  rsyslog
 [ - ]  console-setup.sh         [ - ]  saned
 [ + ]  cron                     [ + ]  sendmail
 [ + ]  cups                     [ + ]  speech-dispatcher
 [ + ]  cups-browsed             [ - ]  spice-vdagent
 [ + ]  dbus                     [ + ]  ssh
 [ - ]  dns-clean                [ + ]  sysstat
 [ + ]  gdm3                     [ - ]  thermald
 [ + ]  grub-common              [ + ]  udev
 [ - ]  hwclock.sh               [ + ]  ufw
 [ + ]  irqbalance               [ + ]  unattended-upgrades
 [ + ]  kerneloops               [ - ]  uuidd
 [ - ]  keyboard-setup.sh        [ + ]  whoopsie
 [ + ]  kmod                     [ - ]  x11-common

In the display above:

+ = active
- = inactive
? = no status option available

Here's a useful command for getting a quick report on disk space usage:

$ journalctl --disk-usage
Archived and active journals take up 824.1M in the file system.

If you want to focus on a particular process, you can do that by providing a PID (truncated) as in the example below.

$ journalctl _PID=787
-- Logs begin at Mon 2018-05-14 15:16:11 EDT, end at Tue 2018-10-16 08:25:17 EDT
Aug 03 18:02:46 butterfly apport[787]:  * Starting automatic crash report genera
Aug 03 18:02:46 butterfly apport[787]:    ...done.
-- Reboot --
Sep 16 13:26:34 butterfly atopacctd[787]: Version: 2.3.0 - 2017/03/25 09:59:59
Sep 16 13:26:34 butterfly atopacctd[787]: accounting to /run/pacct_source
-- Reboot --
Oct 03 18:08:41 butterfly apport[787]:  * Starting automatic crash report genera
Oct 03 18:08:41 butterfly apport[787]:    ...done.
-- Reboot --
Oct 15 14:07:11 butterfly snapd[787]: AppArmor status: apparmor is enabled but s
Oct 15 14:07:12 butterfly snapd[787]: AppArmor status: apparmor is enabled but s
Oct 15 14:07:12 butterfly snapd[787]: daemon.go:344: started snapd/2.35.2 (serie
Oct 15 14:07:12 butterfly snapd[787]: autorefresh.go:376: Cannot prepare auto-re

NOTE: The systemd journal's configuration file is /etc/systemd/journald.conf.

Wrap-up

The variety of log files on Linux systems is somewhat overwhelming, but discovering a handful of commands that can help pinpoint problems can save you a lot of time and stress.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:
Now read: Getting grounded in IoT