Making good use of the files in /proc

The /proc file system first made its way into some Unix operating systems (such as Solaris) in the mid-1990s, promising to give users more and easier access into the kernel and to running processes. It was a very welcome enhancement — looking and acting like a regular file system, but delivering hooks into the kernel and the ability to treat processes as files. It went well beyond what we could do with ps and other common commands for examining processes and the system they run on.

When it first appeared, /proc took a lot of us by surprise. We were used to devices as files, but access to processes as files was new and exciting. In the years since, /proc has become more of a go-to source for process information, but it retains an element of mystery because of the incredible detail that it provides.

+ Also on Network World: All you need to know about Unix environment variables +

When you cd over to /proc, the most obvious difference between it and regular file systems is that almost all but a handful of files show up as having a length of 0 — even though you can use commands such as cat to extract quite a lot of very useful information from these “empty” files. Some of these files even allow kernel variables to be changed, and some can be a bit overwhelming.

There are two broad categories of data available in /proc — process-specific data that resides in a series of directories with names that reflect the process IDs and files that represent the state of the system.

$ cd /proc
$ ls
1  1345 16   1788 2172 58  740  dma     pagetypeinfo
10 1351 1608 1793 218  59  75   driver  partitions
...

Clearly, the numeric entries relate to processes and are directories, while the other entries are largely files (some are directories) related to the kernel and the system as a whole.

System data

The system files provide extensive information on the state of the server, some of which is easily accessible using other commands. Using /proc, however, it’s assembled in one location and available using commands such as cat and more. Here’s an example:

$ cat partitions
major minor  #blocks  name

   8        0   78150744 sda
   8        1   39061504 sda1
   8        2    9765888 sda2
   8        3   29322240 sda3

The full collection of files can be used to answer a lot of questions about your system. For example, where is your swap partition?

$ cat swaps
Filename                Type            Size    Used    Priority
/dev/sda2               partition       9765884 129104  -1

What is the load average right now?

$ cat loadavg
0.05 0.01 0.00 1/385 13087

That’s more numbers than you might expect to see when looking at load averages, but here’s what they represent. The three are the usual one-, five- and fifteen-minute run queue sizes — like you would see with uptime and top. The fourth number (1/385) shows both the number of currently runnable kernel processes and threads (before the slash) and the number of kernel scheduling entities that currently exist on the system (after the slash). The fifth field shows the PID of the most recently created process.

Now, let’s look at uptime.

$ cat uptime
175361.43 348536.36
$ cat uptime
175370.00 348553.47

The unusual numbers from the uptime file represent the total numbers of seconds that the system has been up. In this case, it’s a little more than 48 hours. The second is the number of seconds the system has been idle. If you look closely, you’ll notice that the second number is actually larger than the first. How can the system have spent more time idle than it’s been up? Good question! The answer is that given a quad-core CPU, idle time can theoretically be four times larger than up time.

Now, let’s check out the CPU.

$ head cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Duo CPU     E6550  @ 2.33GHz
stepping        : 11
microcode       : 0xb3
cpu MHz         : 2327.664
cache size      : 4096 KB
physical id     : 0

The meminfo file provides details about the system’s memory. It shows how much memory is on the system, how much is free and available, etc.

$ head meminfo
MemTotal:        2003388 kB
MemFree:          216856 kB
MemAvailable:    1044432 kB
Buffers:           70276 kB
Cached:           750864 kB
SwapCached:         6840 kB
Active:           704584 kB
Inactive:         666552 kB
Active(anon):     303332 kB
Inactive(anon):   384776 kB

The version file describes the OS in a manner similar to the uname -a command.

$ cat version
Linux version 4.4.0-96-generic (buildd@lgw01-10) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017

Some of the files will require quite a bit of analysis. We can see information on CPUs and interrupts in the stat file.

$ cat stat
cpu  59606 4200 22684 35016622 131487 0 402 0 0 0
cpu0 29575 2500 11597 17535235 39583 0 8 0 0 0
cpu1 30030 1700 11086 17481387 91904 0 394 0 0 0
intr 5320389 60 3 0 0 0 0 0 1 1 3 0 0 4 0 0 0 112 374 358871 0 0 0 2 2 0 0 512895 957 17 548 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 12807759
btime 1505998175
processes 13205
procs_running 1
procs_blocked 0
softirq 8145883 11 2931100 35268 513879 356769 0 61865 2885333 0 1361658

The collection of system files covers a wide range of system data. Each is described in the man page for proc (man proc). Some cannot be read except by privileged accounts. Some are directories that contain a series of detailed files.

acpi       execdomains  kpagecgroup   partitions     timer_list
asound     fb           kpagecount    sched_debug    timer_stats
buddyinfo  filesystems  kpageflags    schedstat      tty
bus        fs           loadavg       scsi           uptime
cgroups    interrupts   locks         self           version
cmdline    iomem        mdstat        slabinfo       version_signature
consoles   ioports      meminfo       softirqs       vmallocinfo
cpuinfo    irq          misc          stat           vmstat
crypto     kallsyms     modules       swaps          zoneinfo
devices    kcore        mounts        sys
diskstats  keys         mtrr          sysrq-trigger
dma        key-users    net           sysvipc
driver     kmsg         pagetypeinfo  thread-self

Process data

The files that represent individual processes are stored in /proc/### directories where ### represents the process ID of each running process. As you might suspect, some of these will come and go very quickly, especially on a very busy system.

The comm file shows the command being run. In this case, it’s a script.

$ cd /proc/13325
$ cat comm
runme

The cmdline file shows the same information with a little more detail. The example commands below display the content in two ways.

$ cat cmdline
/bin/bash./runme$
$ od -bc cmdline
0000000 057 142 151 156 057 142 141 163 150 000 056 057 162 165 156 155
          /   b   i   n   /   b   a   s   h