Monitoring paging on UNIX systems

UNIX provides a system command, vmstat, to help you monitor system activity such as paging. The abbreviated command syntax is:

vmstat interval count

The interval is the time between rows of output, and count is the number times a row of output is displayed. For more information about vmstat (including its options and field descriptions), see your operating system's documentation. Here is an example:

>  vmstat 2 3
procs    memory            page            disk     faults     cpu
r b w swap    free  re mf pi po fr de sr s0 s1 sd  in   sy  cs us sy id

0 0 0 3312376 31840 0  8  0  0  0  0  0   0  0  0 297  201 472 82  4 14
0 0 0 3312376 31484 2  3  0  0  0  0  0   0  0  0 260  169 597 80  3 17
0 0 0 3312368 31116 0  8  0  0  0  0  0   0  0  0 205 1202 396 67  4 29

The above output shows a steady Sybase IQ querying state where the physical memory of the machine has not been overallocated. Little to no system page faulting is occurring. These next set of examples show vmstat output that indicates a problem. (The output shown omits some of the above fields to fit better on the page.)

procs     memory            page               faults       cpu
r b w swap    free   re   mf  pi po fr de sr  in   sy   cs us sy  id

0 0 0 217348  272784  0  148  11  3  9  0  2 251 1835  601  6  3  91
0 0 0 3487124 205572  0    5   0  0  0  0  0  86  131  133  0  1  99
0 0 0 3487124 205572  0    5   0  0  0  0  0  71  162  121  0  0 100
0 0 0 3483912 204500  0  425  36  0  0  0  0 169  642  355  2  2  96
0 0 0 3482740 203372  0   17   6  0  0  0  0 158  370  210  1  3  97
0 0 0 3482676 203300  0    4  10  0  0  0  0 160 1344  225  1  2  97
0 0 0 3343272 199964  1 2123  36  0  0  0  0 213  131  399  7  8  85
0 0 0 3343264 185096  0  194  84  0  0  0  0 283  796  732  1  6  93
0 0 0 3342988 183972  0   17  58  0  0  0  0 276 1051  746  2  4  94
0 0 0 3342860 183632  0  119 314  0  0  0  0 203 1660  529  3  4  94
0 0 0 3342748 182316  2  109 184  0  0  0  0 187  620  488  4  2  95
0 0 0 3342312 181104  2  147  96  0  0  0  0 115  256  260  9  2  89
0 0 0 3340748 179180  0  899  26  0  0  0  0 163  836  531  4  4  92
0 0 0 3328704 167224  0 2993   6  0  0  0  0  82 2195  222  4  7  89

The first line of the above output provides a summary of the system activity after the machine was started. The first three lines show that there is approximately 200MB of free physical memory and that the machine is idle. The fourth line corresponds to Sybase IQ starting up for the first time. Beginning at the eighth line, the amount of free memory starts to reduce rapidly. This corresponds to the Sybase IQ buffer caches being allocated and database pages being read in from disk (note that CPU usage has increased). At this time there is little user CPU time as no queries have begun.

procs     memory                page             faults      cpu
r  b w swap    free   re mf  pi   po fr de sr in    sy  cs   us sy id

 7 0 0 3247636 58920  0 1880 1664 0  0  0  0  1131  442 1668 80 18  3
18 0 0 3246568 43732  0  709 1696 0  0  0  0  1084  223 1308 90 10  1
12 0 0 3246604 37004  0  358  656 0  0  0  0   600  236  722 95  5  0
15 0 0 3246628 32156  0  356 1606 0  0  0  0  1141  226 1317 91  9  0
19 0 0 3246612 26748  0  273 1248 0  0  0  0   950  394 1180 92  7  0

The above output is from slightly later when the query is underway. This is evident from the user mode CPU level (us field). The buffer cache is not yet full as page-in faults (pi field or KB paged in) are still occurring and the amount of free memory is still going down.

procs     memory            page                    faults        cpu
r  b w swap    free  re mf pi   po  fr   de sr  in   sy   cs   us sy id

21 0 0 3246608 22100 0 201 1600   0    0 0    0 1208 1257 1413 88 12  0
18 0 0 3246608 17196 0 370 1520   0  464 0  139  988  209 1155 91  8  0
11 0 0 3251116 16664 0 483 2064 138 2408 0  760 1315  218 1488 88 12  0
30 0 0 3251112 15764 0 475 2480 310 4450 0 1432 1498  199 1717 87 13  0

The above output is from even later. On the third line of the output it shows that the system has reached its threshold for the amount of free memory it can maintain. At this point, page-outs (po field or KB paged out) occur and the level of system mode CPU (sy field) increases accordingly. This situation results because physical memory is overallocated: the Sybase IQ buffer caches are too big for the machine. To resolve this problem, reduce the size of one or both of the buffer caches.