Tools to break apart large log files.

[Chris F.A. Johnson]

yOn Wed, 9 Dec 2009, Scott Sullivan wrote:

> The log files have a time stamp at the front of each line, like this:
> 
> [2009/12/09 19:10]  UserA: Text
> [2009/12/09 19:11]  UserB: Responce
> [2009/12/09 19:11]  UserB: More Text
> [2009/12/09 19:11]  UserA: ??? Profit
> 
> I want something that will register when the gap between the time stamps is
> greater then a given value and split them out to separate files.


secs=123456 ## max number of seconds between lines
awk -v secs=$secs -F'[][]' '{
 split($2,dt," ")
 split(dt[1],d,"/")
 split(dt[2],t,":")
 stamp = (d[1] * 365.25 + d[2] * 30.4375 + d[3]) * 86400 + t[1] * 60 + t[2]
}

(stamp - laststamp) > secs { ofile = "outfile" ++n }

{
 print > ofile
 laststamp = stamp
}

' "$file"


> On 12/09/2009 10:14 PM, Chris F.A. Johnson wrote:
> > On Wed, 9 Dec 2009, Scott Sullivan wrote:
> > 
> >    
> > > What I Have:
> > >      Chat log files that span multiple months (like IRC channel logs).
> > > 
> > > What I want:
> > >      To break those large files into many smaller files based on a certain
> > > time
> > > gag between posts.
> > >      ie. Break to new file IF time between lines n and n+1 is>  2 hour.
> > >      
> >     Use awk. Without seeing what the files look like, I can't be more
> >     specific.
> > 
> >    
> --
> The Toronto Linux Users Group.      Meetings: http://gtalug.org/
> TLUG requests: Linux topics, No HTML, wrap text below 80 columns
> How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists
> 

-- 
   Chris F.A. Johnson, webmaster         <http://woodbine-gerrard.com>
   ===================================================================
   Author:
   Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
   Pro Bash Programming: Scripting the GNU/Linux Shell (2009, Apress)
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists