Making good use of the files in /proc

The /proc file system provides amazingly detailed views into your processes and your system's guts.

Making good use of the files in /proc
Windell Oskay (CC BY 2.0)

The /proc file system first made its way into some Unix operating systems (such as Solaris) in the mid-1990s, promising to give users more and easier access into the kernel and to running processes. It was a very welcome enhancement — looking and acting like a regular file system, but delivering hooks into the kernel and the ability to treat processes as files. It went well beyond what we could do with ps and other common commands for examining processes and the system they run on.

When it first appeared, /proc took a lot of us by surprise. We were used to devices as files, but access to processes as files was new and exciting. In the years since, /proc has become more of a go-to source for process information, but it retains an element of mystery because of the incredible detail that it provides.

+ Also on Network World: All you need to know about Unix environment variables +

When you cd over to /proc, the most obvious difference between it and regular file systems is that almost all but a handful of files show up as having a length of 0 — even though you can use commands such as cat to extract quite a lot of very useful information from these “empty” files. Some of these files even allow kernel variables to be changed, and some can be a bit overwhelming.

There are two broad categories of data available in /proc — process-specific data that resides in a series of directories with names that reflect the process IDs and files that represent the state of the system.

$ cd /proc
$ ls
1  1345 16   1788 2172 58  740  dma     pagetypeinfo
10 1351 1608 1793 218  59  75   driver  partitions
...

Clearly, the numeric entries relate to processes and are directories, while the other entries are largely files (some are directories) related to the kernel and the system as a whole.

System data

The system files provide extensive information on the state of the server, some of which is easily accessible using other commands. Using /proc, however, it’s assembled in one location and available using commands such as cat and more. Here's an example:

$ cat partitions
major minor  #blocks  name

   8        0   78150744 sda
   8        1   39061504 sda1
   8        2    9765888 sda2
   8        3   29322240 sda3

The full collection of files can be used to answer a lot of questions about your system. For example, where is your swap partition?

$ cat swaps
Filename                Type            Size    Used    Priority
/dev/sda2               partition       9765884 129104  -1

What is the load average right now?

$ cat loadavg
0.05 0.01 0.00 1/385 13087

That’s more numbers than you might expect to see when looking at load averages, but here’s what they represent. The three are the usual one-, five- and fifteen-minute run queue sizes — like you would see with uptime and top. The fourth number (1/385) shows both the number of currently runnable kernel processes and threads (before the slash) and the number of kernel scheduling entities that currently exist on the system (after the slash). The fifth field shows the PID of the most recently created process.

Now, let's look at uptime.

$ cat uptime
175361.43 348536.36
$ cat uptime
175370.00 348553.47

The unusual numbers from the uptime file represent the total numbers of seconds that the system has been up. In this case, it’s a little more than 48 hours. The second is the number of seconds the system has been idle. If you look closely, you’ll notice that the second number is actually larger than the first. How can the system have spent more time idle than it’s been up? Good question! The answer is that given a quad-core CPU, idle time can theoretically be four times larger than up time.

Now, let's check out the CPU.

$ head cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Duo CPU     E6550  @ 2.33GHz
stepping        : 11
microcode       : 0xb3
cpu MHz         : 2327.664
cache size      : 4096 KB
physical id     : 0

The meminfo file provides details about the system’s memory. It shows how much memory is on the system, how much is free and available, etc.

$ head meminfo
MemTotal:        2003388 kB
MemFree:          216856 kB
MemAvailable:    1044432 kB
Buffers:           70276 kB
Cached:           750864 kB
SwapCached:         6840 kB
Active:           704584 kB
Inactive:         666552 kB
Active(anon):     303332 kB
Inactive(anon):   384776 kB

The version file describes the OS in a manner similar to the uname -a command.

$ cat version
Linux version 4.4.0-96-generic (buildd@lgw01-10) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017

Some of the files will require quite a bit of analysis. We can see information on CPUs and interrupts in the stat file.

$ cat stat
cpu  59606 4200 22684 35016622 131487 0 402 0 0 0
cpu0 29575 2500 11597 17535235 39583 0 8 0 0 0
cpu1 30030 1700 11086 17481387 91904 0 394 0 0 0
intr 5320389 60 3 0 0 0 0 0 1 1 3 0 0 4 0 0 0 112 374 358871 0 0 0 2 2 0 0 512895 957 17 548 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 12807759
btime 1505998175
processes 13205
procs_running 1
procs_blocked 0
softirq 8145883 11 2931100 35268 513879 356769 0 61865 2885333 0 1361658

The collection of system files covers a wide range of system data. Each is described in the man page for proc (man proc). Some cannot be read except by privileged accounts. Some are directories that contain a series of detailed files.

acpi       execdomains  kpagecgroup   partitions     timer_list
asound     fb           kpagecount    sched_debug    timer_stats
buddyinfo  filesystems  kpageflags    schedstat      tty
bus        fs           loadavg       scsi           uptime
cgroups    interrupts   locks         self           version
cmdline    iomem        mdstat        slabinfo       version_signature
consoles   ioports      meminfo       softirqs       vmallocinfo
cpuinfo    irq          misc          stat           vmstat
crypto     kallsyms     modules       swaps          zoneinfo
devices    kcore        mounts        sys
diskstats  keys         mtrr          sysrq-trigger
dma        key-users    net           sysvipc
driver     kmsg         pagetypeinfo  thread-self

Process data

The files that represent individual processes are stored in /proc/### directories where ### represents the process ID of each running process. As you might suspect, some of these will come and go very quickly, especially on a very busy system.

The comm file shows the command being run. In this case, it’s a script.

$ cd /proc/13325
$ cat comm
runme

The cmdline file shows the same information with a little more detail. The example commands below display the content in two ways.

$ cat cmdline
/bin/bash./runme$
$ od -bc cmdline
0000000 057 142 151 156 057 142 141 163 150 000 056 057 162 165 156 155
          /   b   i   n   /   b   a   s   h  \0   .   /   r   u   n   m
0000020 145 000
          e  \0
0000022

The sched file shows process statistics with very precise timing details and information on how the particular process is running, including context switches, blocking and wait statistics.

$ head sched
runme (13325, #threads: 1)
-------------------------------------------------------------------
se.exec_start                                :     176978922.265960
se.vruntime                                  :           121.131898
se.sum_exec_runtime                          :             4.165772
se.statistics.sum_sleep_runtime              :        220026.189448
se.statistics.wait_start                     :             0.000000
se.statistics.sleep_start                    :     176978922.265960
se.statistics.block_start                    :             0.000000
se.statistics.sleep_max                      :         20017.912123

The status file shows the current process state (sleeping), process and parent process IDs, the UID and GIDs associated with the process.

$ head status
Name:   runme
State:  S (sleeping)
Tgid:   13325
Ngid:   0
Pid:    13325
PPid:   13274
TracerPid:      0
Uid:    1004    1004    1004    1004
Gid:    1001    1001    1001    1001
FDSize: 256

We can also examine the limits associated with a particular process.

$ cat limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             7666                 7666                 processes
Max open files            1024                 1048576              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       7666                 7666                 signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Wrap-up

There's a lot of data available in /proc that details how the system is running and how individual processes are performing. Some of this information is easy to understand and mimics the output we might see from various commands. Other data is very detailed but allows programmers as well as sysadmins to delve into precise details of how processes are behaving and how the system is managing its load.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:
Must read: 10 new UI features coming to Windows 10