Efficent handling of sockets

[Lennart Sorensen]

On Fri, Feb 20, 2009 at 11:35:22AM -0500, D. Hugh Redelmeier wrote:
> Process switching is expensive in Linux.  Why would you say otherwise?
> Not the fault of Linux, it is the nature of modern hardware.  The
> semantics of "process" could be pared down to allow a cheaper
> implementation but we'd not like that semantics (eg. no process
> isolation).
> 
> I suggested epoll so that bitstrings did not have to be copied in and
> out of kernel space at each "get next event".  So the comparision is
> not relevant.  (I admit that I've never used epoll.)
> 
> Let's assume that we were using select.  How many file descriptors do
> you think we would be considering?  How many threads?  Should they be
> the same order of magnitude?

I like one thread per socket myself.  Simple and clean.

> How many threads can a Linux kernel handle well?  I have no idea, so
> this isn't rhetorical.

Quite a lot these days.

> How about 1000 of each?  Then each socket call needs to copy in and
> copy out 125 contiguous bytes.  Yuck.  But I bet that is a lot less
> than context switching (between user processes (same as between user
> threads)).  How much process context is in registers?
> 8 * 8B plain registers (roughly)
> 8 * 8B FP registers (guess)

Yeah those.  On CPU instruction does that.  How many instructions does
it take to copy 125 bytes from user space to kernel space?  The registers
happen all in kernel space.

Thread switches are much less expensive than process switches.
They haven't always been, but for the last number of years they have been.
(On Linux at least.  Many other OSs have much higher context switch costs)

> unknown MMU context, some of which causes caching hickups.

No MMU changes when switching threads.  And if you are swithching to
another process, then it isn't related to the problem and there is
nothing you can do about it.

> TSC stuff?

No that's CPU, not process specific.

> I don't know what else.
> 
> Remember, under load, the select version should handle multiple
> transactions per select.  Not true of the analogous thread version.

But it only handles one at a time, even if it doesn't have to call
select again.

> Locks in userland code (or their lack!) seem to be a great source of
> problems too.

If the different sockets have to share resources, then locks can be a
problem, but serializing the whole job isn't going to make it faster.

> The existence of thousands of sockets might be a sign of a bad design
> too.  Perhaps the demultiplexing is being done in the wrong place.  It
> depends on what is being attempted.

Certainly something that should be considered too.

> How so?  Are you saying that you have thousands of cores?

No, but I also didn't think all 1000 established sockets were likely to
deliver data at the same time.

Certainly 4 cores is common, 8 and even 16 cores is becoming quite common,
and sun has systems with 64 cores per chip.

> If you are handling thousands of transaction streams, you had better
> think about scheduling.

Sure, but it may be that the kernel scheduler still works better.

-- 
Len Sorensen
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists