Brainstorming help needed (Data copy job - ideas needed)
Madison Kelly
linux-5ZoueyuiTZhBDgjK7y7TUQ at public.gmane.org
Tue Jun 8 15:18:43 UTC 2004
Lennart Sorensen wrote:
> Assuming your file list is created in a deterministic (and consistent)
> manner I wouldn't worry about it, since the only files that would move
> from slice1 to slice2 would be the last files that went to slice1 in the
> case where new files have been inserted in the list of files before
> them, hence shifting the end of slice1 to slice2. Unless you expect
> many insertions of new files or potentially many deletes (which would
> shift data back from slice2 to slice1) very few of your files would not
> end up in the same location as the last backup. For the few that do
> move, oh well. Probably not worth making anything complicated to deal
> with it.
>
> If the file list comes out random each time it is generated, perhaps you
> ought to create a better file list gnerator. :)
>
> Any good reason to have the backup going to multiple places?
Hi Lennart, thanks for the feedback!
The reason for more than one backup is to let people with a lot of data
(size-wise) spread the backup job over multiple destinations
simultaneously. So say you have 1TB of data to backup you could get five
backup drive chassis and groups of five 200GB drives and run a complete
backup at once.
The file list generator will run pretty much the same each time, baring
big changes. I imagine for 90% of the people who use this is will work
great... I was trying to figure how to best server the other 10% with
funkier setups. I ran it through my head over the night and I think I
have an idea of how to go about it now. Once worry I had was how to
handle running out of disk space and backup jobs whose source data was
larger than the destination media.
I think what I will do above what I described already is have the user
define a timeout period for how long to wait for new destination media
when they schedule a job. When the media fills up it will note which
files made it, record them to the backup log and then dismount the
partition and e-mail the user (defined in the config file) that it is
time to swap the media. Then every two minutes (or so, defined by the
config) rescan for new destination media and when it is found mount it,
write a new backup list for that partition which includes the missing
file (or as much as it can) and then start the backup. If it runs out of
space again, repeat.
This should allow for suddenly running out of space and spanning disks.
I hope... ;)
Madison
--
The Toronto Linux Users Group. Meetings: http://tlug.ss.org
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://tlug.ss.org/subscribe.shtml
More information about the Legacy
mailing list