Efficent handling of sockets

[D. Hugh Redelmeier]

| From: Lennart Sorensen <lsorense-1wCw9BSqJbv44Nm34jS7GywD8/FfD2ys at public.gmane.org>

| On Thu, Feb 19, 2009 at 06:24:52PM -0500, D. Hugh Redelmeier wrote:
| > Threads are a very general purpose mechanism and are implemented quite
| > expensively compared with a reasonable "get next event" loop.
| 
| Threads are not expensive in linux.  I wouldn't be surprised if the
| overhead of managing bitstrings in select costs more than threads.

Process switching is expensive in Linux.  Why would you say otherwise?
Not the fault of Linux, it is the nature of modern hardware.  The
semantics of "process" could be pared down to allow a cheaper
implementation but we'd not like that semantics (eg. no process
isolation).

I suggested epoll so that bitstrings did not have to be copied in and
out of kernel space at each "get next event".  So the comparision is
not relevant.  (I admit that I've never used epoll.)

Let's assume that we were using select.  How many file descriptors do
you think we would be considering?  How many threads?  Should they be
the same order of magnitude?

How many threads can a Linux kernel handle well?  I have no idea, so
this isn't rhetorical.

How about 1000 of each?  Then each socket call needs to copy in and
copy out 125 contiguous bytes.  Yuck.  But I bet that is a lot less
than context switching (between user processes (same as between user
threads)).  How much process context is in registers?
8 * 8B plain registers (roughly)
8 * 8B FP registers (guess)
unknown MMU context, some of which causes caching hickups.
TSC stuff?
I don't know what else.

Remember, under load, the select version should handle multiple
transactions per select.  Not true of the analogous thread version.

| Because I like software that tries to scale, and single threaded processes
| using select on a huge number of sockets doesn't scale.  I also think
| the select interface is awful.

Locks in userland code (or their lack!) seem to be a great source of
problems too.

The existence of thousands of sockets might be a sign of a bad design
too.  Perhaps the demultiplexing is being done in the wrong place.  It
depends on what is being attempted.

| On a multi core system (as most are) you wouldn't even have to context
| switch (which is very cheap on linux too)

How so?  Are you saying that you have thousands of cores?

| > If some (but not all) transactions are expensive to process, they can
| > then be farmed off to other threads or processes.
| 
| Which just means you are writing a scheduler to some extent, which is
| something the kernel already has, and its scheduler is often better than
| one thrown together without much thought into the design.

If you are handling thousands of transaction streams, you had better
think about scheduling.
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists