Most admins monitoring system performace with sar (sysstat) collect performance data over the span of a week or two, but you can easily extend this to four weeks. In fact, you can extend the coverage period to well beyond a month if you need to maintain a longer view of performance and if you have enough disk space to hold all the data. Today’s post details how to configure the coverage period and provides a script that will provide daily averages for as many days’ of /var/log/sa## files you retain.
To change the number of days that sar will maintain data files, go to the /etc/sysconfig directory and edit the sysstat file. Change HISTORY=7 to HISTORY=28. If you change that number to anything greater than 28, you will end up with your sa## files separated into folders by month.
# How long to keep log files (days), maximum is a month HISTORY=7
If you keep more than 28 days of performance data, sar will create a directory for each month. If just started collecting data with HISTORY set to greater than 28, you would see something like this.
drwxr-xr-x. 2 root root 4096 Aug 19 19:11 201208 lrwxrwxrwx. 1 root root 11 Aug 19 19:11 sa19 -> 201208/sa19
Notice that a directory has been set up for August 2012 in year/month format. A symbolic link associates the current collection (August 19th) to the data file in the 201208 directory. So /var/log/sa/sa19 is still a valid reference.
If you intend to hold performance data on line for a long period of time, take the size of one day’s or one week’s collection and multiply by the number of days or weeks to get the overall size. The data files should all be the same size unless your weekend collections are more sparse.
Today’s script below is similar to one that I presented a couple weeks ago, but it will run against any number of /var/log/sa/sa## files. Plus, it doesn’t care about the day of the week. Instead, it gets the date for each data file from the files themselves and displays these dates in the 08/19/2012 format.
#!/bin/bash
#
# summarize daily performance data
cd /var/log/sa
# ------------
# memory usage
# ------------
echo
echo "Memory & Swap"
echo "============="
echo -n " "
sar -r -f $file | head -3 | tail -1 | awk '{print substr($0,12,100)}'
for file in `ls -tr sa* | grep -v sar`
do
dt=`sar -r -f $file | head -1 | awk '{print $NF}'`
echo -n $dt
sar -r -f $file | tail -1 | sed "s/Average: //"
done
# ------------
# load
# ------------
echo
echo "System Load"
echo "============="
echo -n " "
sar -q -f $file | head -3 | tail -1 | awk '{print substr($0,12,100)}'
for file in `ls -rt sa* | grep -v sar`
do
dt=`sar -q -f $file | head -1 | awk '{print $NF}'`
echo -n $dt
sar -q -f $file | tail -1 | sed "s/Average: //"
done
# ------------
# CPU Usage
# ------------
echo
echo "CPU Usage"
echo "============="
echo -n " "
sar -u -f $file | head -3 | tail -1 | awk '{print substr($0,12,100)}'
for file in `ls -tr sa* | grep -v sar`
do
dt=`sar -u -f $file | head -1 | awk '{print $NF}'`
echo -n $dt
sar -u -f $file | tail -1 | sed "s/Average: //"
done
Here is an example of what the data might look like if we’d been collecting since the beginning of the current month and had data files up to the 8th. In this example, we are only reporting on three types of performance measurements from the data available:
Memory & Swap
=============
kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
08/01/2012 486528 32453580 98.52 291310 16291925 1064044 1033100 49.26 936512
08/02/2012 486497 32453611 98.52 309682 16275590 1064044 1033100 49.26 936512
08/03/2012 487076 32453032 98.52 305501 16282480 1064044 1033100 49.26 936512
08/04/2012 491773 32448335 98.51 312822 16269869 1064044 1033100 49.26 936512
08/05/2012 367462 32572646 98.88 334573 16469293 1064044 1033100 49.26 936512
08/06/2012 422750 32517358 98.72 737239 15490014 1064044 1033100 49.26 936512
08/07/2012 469591 32470517 98.57 352373 16080744 1064044 1033100 49.26 936512
08/08/2012 491786 32448322 98.51 300389 16179809 1064044 1033100 49.26 936512
System Load
=============
runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
08/01/2012 0 365 0.06 0.06 0.05
08/02/2012 0 366 0.07 0.06 0.05
08/03/2012 0 366 0.07 0.07 0.06
08/04/2012 0 366 0.07 0.07 0.06
08/05/2012 1 369 0.47 0.46 0.45
08/06/2012 0 365 0.02 0.02 0.01
08/07/2012 0 366 0.06 0.05 0.04
08/08/2012 0 366 0.06 0.06 0.05
CPU Usage
=============
CPU %user %nice %system %iowait %steal %idle
08/01/2012 all 0.52 0.00 0.06 0.04 0.00 99.38
08/02/2012 all 0.55 0.00 0.06 0.04 0.00 99.35
08/03/2012 all 0.57 0.00 0.06 0.05 0.00 99.32
08/04/2012 all 0.59 0.00 0.06 0.05 0.00 99.30
08/05/2012 all 2.61 0.00 0.17 0.77 0.00 96.45
08/06/2012 all 0.16 0.00 0.04 0.02 0.00 99.79
08/07/2012 all 0.51 0.00 0.06 0.02 0.00 99.42
08/08/2012 all 0.53 0.00 0.06 0.04 0.00 99.38
The script itself is fairly straightforward. While the code is repetitious, I’ve broken it into sections in order to provide headings for each section. In addition, the sar commands within each section call out the different performance measurements — -r for memory stats, -q for load stats, and so on. You could duplicate one of these sections and modify the headings and sar options to add other measurements to your report.
As is, the script will only report on the files in /var/log/sa. If you are collecting more than 28 days’ worth of performance data, you would have to modify this script to look into each directory. That shouldn’t be too hard.
