faster rsync of huge directories

[Nadav Har'El]

On Mon, Apr 12, 2010, Shachar Shemesh wrote about "Re: faster rsync of huge directories":
> Upgrade both ends to rsync version 3 or later. That version starts the 
> transfer even before the file list is completely built.

Maybe I'm missing something, but how does this help?

It may find the first file to copy a little quicker, but finishing the
rsync will take exactly the same time, won't it?
Also, if nothing has changed, it will take it exactly the same time to
figure this out, won't it?

I'm not sure what his problem is, though. Is it the fact that the remote
rsync takes a very long time to walk the huge directory tree, or the fact
that sending the whole list over the network is slow?
If it's the first problem, then maybe switching to a different filesystem,
or reorganizing your directory structure (e.g., not to have more than a few
hundred files per directory) will help.
If it's the second problem, then maybe rsync improvements are due - i.e., to
use rsync's delta protocol not only on the individual files, but also on the
file list.

-- 
Nadav Har'El                        |       Monday, Apr 12 2010, 28 Nisan 5770
nyh at math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Fame: when your name is in everything but
http://nadav.harel.org.il           |the phone book.