What is eating swap?

[Fraser Campbell]

Funny that I haven't really run into this one before, hoping someone here has 
ideas ...

I have a box with lots of free memory where swap usage is growing (slowly) 
despite lots of free memory and swap is actively being used (according to 
vmstat).  Here's what free has to say:

                 total       used       free     shared    buffers     cached
    Mem:       1034640     439260     595380          0      34404      84544
    -/+ buffers/cache:     320312     714328
    Swap:      2097144    1793680     303464

It is RHEL4 (kernel 2.6.9-22.ELsmp 32-bit).

There is no single process whose virtual size wouldn't fit within the 700MB of 
supposedly free memory.  Interestingly although swap use is growing, "used" 
memory (the 320312 figure) is supposedly dropping.

Normally when a box has free memory I can coax it into increasing the 
filesystem cache easily (find /usr -type f -exec cat '{}' \; > /dev/null), in 
this case filesystem cache does not grow at all no matter how much disk I 
read ... this suggests to me that there really is no free memory, or perhaps 
just that the kernel is in a really bad state.

Is there a tool which can dig through memory to say which processes are 
pointing at the swapspace?

Red Hat has an FAQ on this topic and they suggest looking at the swapped field 
in top (http://kbase.redhat.com/faq/FAQ_85_9807.shtm).  As far as I can tell 
they are completely wrong, all of the numbers from top look quite bogus (on 
this and other machines).

My other thought was that the box was hacked and some processes hidden from 
me ... at least so far I don't see any such indication.

-- 
Fraser Campbell <fraser-Txk5XLRqZ6CsTnJN9+BGXg at public.gmane.org>
Georgetown, Ontario, Canada
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists