[GTALUG] System Monitor / sysmon

[D. Hugh Redelmeier]

I have sysmon on my Linux systems.  It is a nice graphical tool to show 
CPU, memory, and network loads.  It produces strip charts.

(It also does other things, but that's not my current concern.)

When I want to figure out what's taking so long, sysmon is a handy tool.

But I think that my system's bottleneck is often disk I/O.  sysmon has no 
strip chart for that.  I don't even know how it should be done:

- each device should probably have a different trace, just like each 
  processor core does

- percentage would be ideal but I don't know what that would mean.  Do you 
  count I/O ops?  Queue length?  Processes waiting?

  + sequential I/O is much faster than random I/O (at least on a real
    disk).

  + Large operations are somewhat slower than small ones (generally
    speaking).

  + output is generally non-blocking (roughly: the write goes into the
    kernel's memory, queued for writing to the device, and the process
    proceeds without waiting to the write to actually complete)

  + input usually blocks: the process issues a read and waits until
    the read finishes before it proceeds

  + Perhaps the simplest metric might be the number of processes
    waiting for disk I/O completion.  But that leaves writes
    unaccounted for.

Do you have any suggestions for a simple (possibly simplistic) tool to
show disk bottlenecks in real time?

PS:

On the system I'm looking at, in the sysmon pane for "Processes",
there is a column for "Disk read total" and another for "Disk write
total".  After a day of running the Libreswan test suite,
gnome-terminal server wins with 3.4GiB of read and 3.9GiB of write.

I wonder what that's all about.  I would not expect Gnome Terminal to
do disk I/O.  As it happens, essentially all the output to the
terminal is being captured by a script(1) command (run within the gnome
terminal session).  It shows 11.4MiB of disk output for one complete run,
which should be half or a third of all that has gone to Gnome Terminal
because I've run the suite two or three times since logging in.
But it is more like a 11.4/3400 or roughly 1/300.

Guess: gnome terminal reads and writes to X count as disk I/O.  That's
pretty messed up.