war story: parallel(1) command

[Walter Dnes]

On Sat, Jul 27, 2013 at 07:30:04PM -0400, Stewart C. Russell wrote
> On 13-07-26 11:03 PM, D. Hugh Redelmeier wrote:
> > 
> > I was previously unaware of the Gnu parallel command.  Pretty neat.
> 
> Handy indeed. I've got this little script that generates shell output to
> convert FLACs to MP3s, but only one at a time. With parallel, I can use
> all of the cores!

  Me three.  I have a similar need batch-converting a couple of hundred
raw files at a time to PNG format and creating additional copies at
1/3rd and 1/4th original size.  I found out "the hard way" that
recklessly forking jobs to the background quickly turned into a
forkbomb.  I then cobbled together a function that looped and consulted
/proc/loadavg before deciding whether to proceed or "sleep 1".  It kept
things from getting totally out of hand, but it wasn't perfect.
parallel let the load climb up to 4 on my 4-core machine, and stayed
there.  It's what I was looking for.

-- 
Walter Dnes <waltdnes-SLHPyeZ9y/tg9hUCZPvPmw at public.gmane.org>
I don't run "desktop environments"; I run useful applications
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists