Brainstorming help needed (Data copy job - ideas needed)

[Madison Kelly]

Lennart Sorensen wrote:
> Assuming your file list is created in a deterministic (and consistent)
> manner I wouldn't worry about it, since the only files that would move
> from slice1 to slice2 would be the last files that went to slice1 in the
> case where new files have been inserted in the list of files before
> them, hence shifting the end of slice1 to slice2.  Unless you expect
> many insertions of new files or potentially many deletes (which would
> shift data back from slice2 to slice1) very few of your files would not
> end up in the same location as the last backup.  For the few that do
> move, oh well.  Probably not worth making anything complicated to deal
> with it.
> 
> If the file list comes out random each time it is generated, perhaps you
> ought to create a better file list gnerator. :)
> 
> Any good reason to have the backup going to multiple places?

Hi Lennart, thanks for the feedback!

The reason for more than one backup is to let people with a lot of data 
(size-wise) spread the backup job over multiple destinations 
simultaneously. So say you have 1TB of data to backup you could get five 
backup drive chassis and groups of five 200GB drives and run a complete 
backup at once.

The file list generator will run pretty much the same each time, baring 
big changes. I imagine for 90% of the people who use this is will work 
great... I was trying to figure how to best server the other 10% with 
funkier setups. I ran it through my head over the night and I think I 
have an idea of how to go about it now. Once worry I had was how to 
handle running out of disk space and backup jobs whose source data was 
larger than the destination media.

I think what I will do above what I described already is have the user 
define a timeout period for how long to wait for new destination media 
when they schedule a job. When the media fills up it will note which 
files made it, record them to the backup log and then dismount the 
partition and e-mail the user (defined in the config file) that it is 
time to swap the media. Then every two minutes (or so, defined by the 
config) rescan for new destination media and when it is found mount it, 
write a new backup list for that partition which includes the missing 
file (or as much as it can) and then start the backup. If it runs out of 
space again, repeat.

This should allow for suddenly running out of space and spanning disks. 
I hope... ;)

Madison
--
The Toronto Linux Users Group.      Meetings: http://tlug.ss.org
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://tlug.ss.org/subscribe.shtml