rsync performance

[Ian Petersen]

On 10/16/07, Ansar Mohammed <ansarm-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> wrote:
> As I understand it, if portions of the file
> are changed, but not the entire file, rsync should only update changed
> portions?

Yes, that's true.  rsync uses cryptographically-strong checksums to
check that local and remote files are the same.

> What's the actual efficiency of this process?

I think it's pretty close to optimal.  My memory of the details is a
little fuzzy, but the basic principal is that the client and server
process each compute "rolling checksums" of the file, sending the sums
back and forth.  Once a checksum is found not to match, the actual
data is sent.  I think there's some provision for each side to
"catch-up" to the other so that, for example, inserting into or
deleting from the middle of the file is still efficient.  There's also
some communication of a checksum of the whole file so that, in the
worst case, rsync degenerates to a simple file transfer plus a bunch
of checksums.

Transformations like encryption and compression both break rsync in
general.  What I mean is that it's more efficient to rsync a large
uncompressed text file than it is to rsync the same file after
compression (assuming some change in the file) because compression
(and encryption) tends to create changes throughout the file even if
the original file only changed a little bit.

The backup that I mentioned earlier that takes place over an 802.11b
network consists of a whole bunch of files that add up to about 9.5
GB.  The daily backup takes about 90 seconds, but there's only about 1
MB worth of changes to the data on a day-to-day basis.  It took almost
2 hours to create the initial image of 2.5 GB, and about 5 hours to
bring it up to 9.5 GB two days later when I changed what was being
backed up.

Ian

PS I just realized that, if you're generally sending data UP a dsl
connection, you're probably limited to something like 800 kb/s, not 5
Mb/s, so you're looking at a transfer time approximately 10 times as
long as I'm experiencing over 802.11b.

-- 
Tired of pop-ups, security holes, and spyware?
Try Firefox: http://www.getfirefox.com
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists