Checking Linux system performance with sar

Sar is a system utility that gives us many ways to examine performance on a Linux system. It provides details on all aspects of system performance including system load, CPU usage, memory use, paging, swapping, disk usage, device load, network activity, etc.

The name “sar” stands for “system activity report,” and it can display current performance, provide reports that are based on log files stored in your system’s /var/log/sa (or /var/log/sysstat) folder, or be set up to automatically produce daily reports. It’s part of sysstat – a collection of system performance monitoring tools.

To check if sar is available on your system, run a command like this:

$ which sar
/usr/bin/sar

If not, you will need to install it with a command like “yum install sysstat” or “apt install sysstat”. You will then be able to run commands like these which collect performance details from your system:

$ sar -d 5 2
Linux 5.11.15-300.fc34.x86_64 (dragonfly)   06/16/2021   _x86_64_      (2 CPU)

12:38:42 PM     DEV     tps   rkB/s   wkB/s   dkB/s areq-sz  aqu-sz   await   %util
12:38:47 PM     sda    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
12:38:47 PM     sr0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
12:38:47 PM     sdb    0.40    0.00    6.40    0.00   16.00    0.01    8.50    0.34
12:38:47 PM     sdc    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
12:38:47 PM     zram0  0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

12:38:47 PM     DEV     tps   rkB/s   wkB/s   dkB/s areq-sz  aqu-sz   await   %util
12:38:52 PM     sda    2.40    0.00  388.80    0.00  162.00    0.01    2.00    0.34
12:38:52 PM     sr0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
12:38:52 PM     sdb    1.40    0.00   21.60    0.00   15.43    0.01    2.00    0.28
12:38:52 PM     sdc    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
12:38:52 PM     zram0  0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

Average:        DEV     tps   rkB/s   wkB/s   dkB/s areq-sz  aqu-sz   await   %util
Average:        sda    1.20    0.00  194.40    0.00  162.00    0.00    2.00    0.17
Average:        sr0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
Average:        sdb    0.90    0.00   14.00    0.00   15.56    0.01    3.44    0.31
Average:        sdc    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
Average:      zram0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

Note that this report shows device usage with activity displayed for /dev/sda and /dev/sdb. It includes two 5-second reports plus the averaging. If you delve into the sar man page, you’ll find explanations of each data column. In the above example, the fields included in the above example include:

DEV     the device
tps     Total number of transfers per second
rkB/s   Number of kilobytes read from the device per second
wkB/s   Number of kilobytes written to the device per second
dkB/s   Number of kilobytes discarded for the device per second
areq-sz Average size (in kilobytes) of the I/O requests
aqu-sz  Average queue length of the requests
await   Average  time (in milliseconds) for I/O requests
%util   Percentage of elapsed time during which I/O requests were issued to the device

The next command pulls data on memory usage with three 5-second reports plus the averages.

$ sar -r 5 3
Linux 5.11.15-300.fc34.x86_64 (dragonfly)       06/16/2021      _x86_64_        (2 CPU)

12:09:54 PM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
12:09:59 PM   3089584   5231848    471144      7.75     35944   2257060   2126524     17.50    645168   2021508        48
12:10:04 PM   3089584   5231848    471140      7.75     35944   2257060   2126524     17.50    645168   2021552        48
12:10:09 PM   3089584   5231848    471140      7.75     35944   2257060   2126524     17.50    645168   2021560       144
Average:      3089584   5231848    471141      7.75     35944   2257060   2126524     17.50    645168   2021540        80

Using a wide terminal window will clearly make the details a lot easier to view.

Looking at current performance stats

The sar commands shown above are all using real time data. In other words, sar is collecting the data as directed and displaying it. No log files are involved. Older stats will not be available.

The default display is CPU. So, if you don’t specify an option, you’ll see something like this:

$ sar 10 2
Linux 5.11.15-300.fc34.x86_64 (dragonfly)       06/16/2021      _x86_64_        (2 CPU)

10:17:50 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:18:00 AM     all      0.05      0.00      0.15      0.05      0.00     99.75
10:18:10 AM     all      0.05      0.00      0.20      0.00      0.00     99.75
Average:        all      0.05      0.00      0.18      0.03      0.00     99.75

To save the output from a command like this in a file, you could run the command like this:

$ sar -o sarfile 6 2

Keep in mind that the data file created (sarfile in this example) will not be a text file that you can display with a cat or more command. However, you can display it using a sar command that includes the file name as an argument:

$ sar -f sarfile
Linux 5.11.15-300.fc34.x86_64 (dragonfly)       06/15/2021      _x86_64_        (2 CPU)

10:15:52 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:15:58 AM     all      0.17      0.00      0.17      0.25      0.00     99.42
10:16:04 AM     all      0.08      0.00      0.33      0.00      0.00     99.58
Average:        all      0.12      0.00      0.25      0.12      0.00     99.50

Looking at previous performance stats

You can easily set sar up to run automatically on your system, store its collected stats in daily files, produce daily reports and remove the oldest of the files after a given number of days (default is 28). This allows you to review performance on recent dates and compare what you’re seeing with current stats. This can often help you answers questions like “What’s different today?” — often the first question you’re likely to ask yourself when problems arise.

Once you’ve collected data using sar, you can look at it with sar commands like this command that extracts data from the sa14 data file:

$ sar -f /var/log/sa/sa14 -b
Linux 5.11.15-300.fc34.x86_64 (dragonfly)       06/14/2021      _x86_64_        (2 CPU)

02:54:22 PM  LINUX RESTART      (2 CPU)

03:00:19 PM       tps      rtps      wtps      dtps   bread/s   bwrtn/s   bdscd/s
03:10:19 PM      1.02      0.02      1.00      0.00      0.01     69.05      0.00
03:20:19 PM      1.05      0.02      1.03      0.00      0.01     75.03      0.00
03:30:19 PM      0.99      0.02      0.97      0.00      0.28     74.35      0.00

To do this, you need to first enable sar to run automatically and save performance stats in data files. The process for doing this is slightly different depending on which Linux distribution you are using.

Fedora

On Fedora, you would use commands like these to enable sar to collect daily stats:

# systemctl start sysstat.service
# systemctl enable sysstat.service
# systemctl status sysstat.service

After this, sar will store its data files in the /var/log/sa directory. A couple days after setting it up, you can expect to see something like this:

$ ls -l
total 1096
-rw-r--r--. 1 root root 103292 Jun 14 23:50 sa14   
The numbers in the file names (e.g., 14) represent the day of the month. The three "sa" files (sa14, sa15 and sa16) are the data files for each of the three days and the sar14 and sar15 files are the performance reports for the first two days. On Fedora, you can determine how many days' worth of data will be saved by using this command:
$ grep HISTORY /etc/sysconfig/sysstat
HISTORY=28

Ubuntu
To enable sar on Ubuntu and related systems, edit the /etc/default/sysstat file and change ENABLED="false" to ENABLED="true".
ENABLED="true"

The log files will be stored in /var/log/sysstat.
Viewing daily reports
To view any of the daily sar reports (e.g., sar15), you can simply use commands like more or cat. These files are simple text files.
Once a day has ended (shortly after midnight), a full-day report will be available with a name like "sar15". While files like "sa14" are used to collect the raw data, files like "sar14" are text files that contain the daily reports. They contain quite a lot of performance details.
$ wc -l sar15
4701 sar15
$ more sar15
Linux 5.11.15-300.fc34.x86_64 (dragonfly)     2021-06-15    _x86_64_        (2 CPU)

00:00:19  CPU   %usr  %nice   %sys %iowait  %steal   %irq  %soft %guest %gnice   %idle
00:10:19  all   0.07   0.00   0.12    0.05    0.00   0.01   0.01   0.00   0.00   99.74
00:10:19    0   0.03   0.01   0.05    0.02    0.00   0.00   0.01   0.00   0.00   99.88
00:10:19    1   0.10   0.00   0.20    0.08    0.00   0.01   0.01   0.00   0.00   99.60
00:20:19  all   0.07   0.00   0.10    0.04    0.00   0.00   0.01   0.00   0.00   99.77
00:20:19    0   0.01   0.00   0.02    0.02    0.00   0.00   0.00   0.00   0.00   99.94
00:20:19    1   0.12   0.00   0.18    0.07    0.00   0.01   0.01   0.00   0.00   99.60
00:30:19  all   0.06   0.00   0.10    0.06    0.00   0.01   0.01   0.00   0.00   99.76
…

To extract data from one of the daily data files (not the report files), you would use a command like this using the -f option to specify the data file to be used:
$ sar -f sa15 | head -22
Linux 5.11.15-300.fc34.x86_64 (dragonfly)       06/15/2021      _x86_64_        (2 CPU)

12:00:19 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:19 AM     all      0.07      0.00      0.14      0.05      0.00     99.74
12:20:19 AM     all      0.07      0.00      0.11      0.04      0.00     99.77
12:30:19 AM     all      0.06      0.00      0.12      0.06      0.00     99.76
12:40:19 AM     all      0.06      0.00      0.12      0.05      0.00     99.77
12:50:19 AM     all      0.07      0.00      0.11      0.05      0.00     99.77
01:00:19 AM     all      0.07      0.02      0.12      0.05      0.00     99.74
01:10:19 AM     all      0.07      0.00      0.11      0.05      0.00     99.77
01:20:19 AM     all      0.30      0.00      0.12      0.04      0.00     99.53
01:30:19 AM     all      0.07      0.00      0.11      0.04      0.00     99.77
01:40:19 AM     all      0.07      0.00      0.11      0.03      0.00     99.78
01:50:19 AM     all      0.07      0.00      0.12      0.03      0.00     99.78
02:00:19 AM     all      0.06      0.00      0.12      0.03      0.00     99.78
02:10:19 AM     all      0.06      0.00      0.13      0.04      0.00     99.77
02:20:19 AM     all      0.06      0.00      0.12      0.04      0.00     99.78
02:30:19 AM     all      0.06      0.00      0.12      0.02      0.00     99.79
02:40:19 AM     all      0.07      0.00      0.12      0.03      0.00     99.79
02:50:19 AM     all      0.07      0.94      0.20      0.05      0.00     98.76
03:00:19 AM     all      0.18      0.00      0.12      0.03      0.00     99.67
03:10:19 AM     all      0.06      0.00      0.12      0.04      0.00     99.78

You can specify a start (-s) and end (-e) times in addition to the file name as in this next example.
$ sar -s 11:00 -e 12:00 -f sa15
Linux 5.11.15-300.fc34.x86_64 (dragonfly)       06/15/2021      _x86_64_        (2 CPU)

11:00:19 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:10:19 AM     all      0.06      0.00      0.17      0.05      0.00     99.73
11:20:19 AM     all      0.05      0.00      0.17      0.05      0.00     99.73
11:30:19 AM     all      0.05      0.00      0.17      0.05      0.00     99.73
11:40:19 AM     all      0.18      0.00      0.17      0.05      0.00     99.60
11:50:19 AM     all      0.06      0.02      0.18      0.05      0.00     99.69
Average:        all      0.08      0.00      0.17      0.05      0.00     99.70

Options, options and options
You can use sar to display stats related to:

• Paging (sar -B)
• I/O and transfer rates (sar -b)
• Block devices (sar -d)
• Currently mounted file systems(sar -F)
• Huge page utilization(sar -H)
• Interrupts (sar -I)
• Power management (sar -m)
• Network statistics (sar -n)
• Run queue and load average (sar -q)
• Memory usage (sar -r)
• Swap utilization (sar -S)
• CPU utilization (sar -u)
• Status of inode, file and other kernel tables (sar -v)
• Swapping (sar -W)
• Context switches (sar -w)
• TTY device activity (sar -y)

In fact, the easiest way to get a listing of all of these options and what they represent is to ask sar for help. Notice that for some options, like networking, you have a choice of which variety of activity (e.g., TCP or sockets) you want to see.

$ sar –help
Usage: sar [ options ] [  [  ] ]
Main options and reports (report name between square brackets):
	-B	Paging statistics [A_PAGE]
	-b	I/O and transfer rate statistics [A_IO]
	-d	Block devices statistics [A_DISK]
	-F [ MOUNT ]
		Filesystems statistics [A_FS]
	-H	Hugepages utilization statistics [A_HUGE]
	-I {  | SUM | ALL }
		Interrupts statistics [A_IRQ]
	-m {  [,...] | ALL }
		Power management statistics [A_PWR_...]
		Keywords are:
		CPU	CPU instantaneous clock frequency
		FAN	Fans speed
		FREQ	CPU average clock frequency
		IN	Voltage inputs
		TEMP	Devices temperature
		USB	USB devices plugged into the system
	-n {  [,...] | ALL }
		Network statistics [A_NET_...]
		Keywords are:
		DEV	Network interfaces
		EDEV	Network interfaces (errors)
		NFS	NFS client
		NFSD	NFS server
		SOCK	Sockets	(v4)
		IP	IP traffic	(v4)
		EIP	IP traffic	(v4) (errors)
		ICMP	ICMP traffic	(v4)
		EICMP	ICMP traffic	(v4) (errors)
		TCP	TCP traffic	(v4)
		ETCP	TCP traffic	(v4) (errors)
		UDP	UDP traffic	(v4)
		SOCK6	Sockets	(v6)
		IP6	IP traffic	(v6)
		EIP6	IP traffic	(v6) (errors)
		ICMP6	ICMP traffic	(v6)
		EICMP6	ICMP traffic	(v6) (errors)
		UDP6	UDP traffic	(v6)
		FC	Fibre channel HBAs
		SOFT	Software-based network processing
	-q [  [,...] | PSI | ALL ]
		System load and pressure-stall statistics
		Keywords are:
		LOAD	Queue length and load average statistics [A_QUEUE]
		CPU	Pressure-stall CPU statistics [A_PSI_CPU]
		IO	Pressure-stall I/O statistics [A_PSI_IO]
		MEM	Pressure-stall memory statistics [A_PSI_MEM]
	-r [ ALL ]
		Memory utilization statistics [A_MEMORY]
	-S	Swap space utilization statistics [A_MEMORY]
	-u [ ALL ]
		CPU utilization statistics [A_CPU]
	-v	Kernel tables statistics [A_KTABLES]
	-W	Swapping statistics [A_SWAP]
	-w	Task creation and system switching statistics [A_PCSW]
	-y	TTY devices statistics [A_SERIAL]

That’s a LOT of choices! You can, however, use the -A option to collect the data provided by all of the options included included in the string “bBdFHSvwWy”.

Wrap-Up

While sar is easy to install and use, it provides a lot of valuable details on how your system is performing and might help you drill down to the cause of a problem when one arises.

The only “problem” with sar is that it generates a LOT of data and you’ll have to become accustomed to using it and familiar with what routine performance on your system looks like before unusual performance or reasons for concern will jump out at you.