CentOS NFS problem -- 30 seconds delay for "cat file"

[Ben Walton]

On Thu, Apr 25, 2013 at 9:39 PM, William Park <opengeometry-FFYn/CNdgSA at public.gmane.org> wrote:
> On Tue, Apr 23, 2013 at 08:17:18PM -0700, DAVID CHIPMAN wrote:
>> William,
>>
>> Can we please see some strace output on the client side when you try
>> to use cat to access a file on an NFS share? Please iuse the "-T"
>> option for strace to get the time spent in system calls. Thanks,
>
> There is nothing revealing about "strace".  It's taking all of its 30sec
> at open(2) call... waiting for something.
>
>     $ mount 192.168.2.3:/home /mnt -o ro
>     $ strace -T cat /mnt/nfs/william/user-agent
>     ...
>     open("/mnt/nfs/home/william/user-agent", O_RDONLY|O_LARGEFILE) = 3 <30.033922>

That's actually a good thing...it means you've eliminated some other
stupid local problem.
>     ...
>
> This is from 32bit CentOS client.  From 64bit machines, it's
>
>     ...
>     open("/mnt/william/user-agent", O_RDONLY) = 3 <30.059668>
>     ...
>
> I got tired of wiping/reinstalling distros to try out.  So, I had to use
> bunch of KVMs.  It took me a while to figure out "bridge" mode so that
> each VM has visible IP, because NFS server rejected connections from
> NAT'ed VM...  something about "illegal port".  Anyways, here are results
> from various clients:
>
>     - Slackware -- instantaneous always
>     - CentOS, Debian, Mint -- always 30sec hang at open(2)
>     - Fedora, OpenSUSE, Ubuntu -- 30sec hang at first attempt to open
>       the file, but immediate on subsequent tries.

These machines may have a caching system enabled...you should be able
to see that fairly easily with strace but also by looking at the
running processes.

>
> At least, I learned about "bridge" mode of QEMU.  This all means that, I
> have to use NFS to locate a file, and use SSH to open it.  Pain, but
> I'll live with it.

Hit the box with wireshark to see where the network hiccup is.  Living
with something so annoying is not the way to go. :)  Compare an ls
with the cat to see if you can isolate the slow nfs rpc's?  There are
also some nfs debugging utilities that would probably give you a good
amount of insight here too.

Thanks
-Ben
-- 
---------------------------------------------------------------------------------------------------------------------------
Take the risk of thinking for yourself.  Much more happiness,
truth, beauty and wisdom will come to you that way.

-Christopher Hitchens
---------------------------------------------------------------------------------------------------------------------------
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists