[GTALUG] Toronto Public Library website

[Alvin Starr]

On 2023-11-08 18:20, Kevin Cozens via talk wrote:
> On 2023-11-08 10:12, Alvin Starr via talk wrote:
>> It does depend on how much you have to recover.
>> I have clients whose data sets take weeks to months to copy over 
>> multi gigabit links.
>
> Not too long ago I had to move a database to a new server. A couple of 
> TBs needed to be moved. Both machines were part of the same network 
> but it still took the better part of a week to copy the data to the 
> new box and get it up and running with the copied data.
A common thing is to move blobs of data out of a database and into files 
on a file system.
Which seems like a good idea till you get a hundred million or so small 
files in a 5-10 deep directory tree.
Then something like a tar or dump can take weeks to run but doing 
something like an image copy of the whole filesystem can be done in a day.
>
>> They did have a verify step in the process where they read the data 
>> after it was written but they only verified that the checksums 
>> matched the data.
>> Fortunately this was found by a customer and none of the financial 
>> institutions involved had a bad day.
>
> Lucky that someone noticed in time to correct the issue. I wonder how 
> a customer was the one to notice it rather than the people doing the 
> data recovery.
>
Someone tried to extract something from a backup and got back a big 
block of zeros.
Once it was figured out what happened it was all hands on deck to fix it 
and get it rolled out to the other customers.

Another one was a clients client who faithfully ran backups each night 
for years.
Just someone forgot to tell them to replace the tape.
You can guess what happened when they had a drive failure......


I am always harping at people to test their recovery process regularly.

-- 
Alvin Starr                   ||   land:  (647)478-6285
Netvel Inc.                   ||   Cell:  (416)806-0133
alvin at netvel.net              ||