bash one-liners with multiple pipes

[Henry Spencer]

On Tue, 11 May 2004, daniel wrote:
> > One problem with any attempt to run real-time output through pipes is that
> > many programs buffer output that's going to a pipe, waiting until they
> > have a full buffer before they actually write it out.  You will still get
> > the same output... eventually... but whether you get it *promptly* depends
> > on little details, and is effectively unpredictable.
> 
> now this is new to me.  i didn't know programs did this sort of thing.

The usual approach -- embedded in the standard libraries for many years --
is that output which is going to a terminal is unbuffered, but output
going anywhere else is buffered.  Unfortunately, processes in the middle
of a pipeline don't know that the output at the *tail* of the pipeline is
going to a terminal! 

> ...the issue seems to come up whenever i 
> pipe a 3rd process ie. "tail | awk | grep..."  but since "tail | grep" seems 
> to work...

As I noted, whether it works or not depends greatly on details.  My guess
would be that "tail -f" is deliberately overriding the library behavior,
so *its* output is going out immediately.  If there's only one more
process in the pipeline, it's doing output to the terminal, so its output
goes out immediately.  But when you add another process in the middle... 

> maybe there's some sort of environment variable i can set to 
> *not* buffer the output?

It would be nice, but I don't know of any such facility (not that I've
gone looking at the innards to make sure).

What you *can* do, possibly at some cost in convenience, is try to roll
all the output processing into a single program.  In particular, note that
awk can do anything that grep and sed can do, so it might be possible (if
a little cumbersome) to just do "tail | awk".

                                                          Henry Spencer
                                                       henry-lqW1N6Cllo0sV2N9l4h3zg at public.gmane.org

--
The Toronto Linux Users Group.      Meetings: http://tlug.ss.org
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://tlug.ss.org/subscribe.shtml