<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=us-ascii" http-equiv="Content-Type">
</head>
<body dir="ltr" bgcolor="#ffffff" text="#000000">
Nadav Har'El wrote:
<blockquote cite="mid:20100412084016.GA27687@fermat.math.technion.ac.il"
type="cite">
<pre wrap="">On Mon, Apr 12, 2010, Shachar Shemesh wrote about "Re: faster rsync of huge directories":
</pre>
<blockquote type="cite">
<pre wrap="">Upgrade both ends to rsync version 3 or later. That version starts the
transfer even before the file list is completely built.
</pre>
</blockquote>
<pre wrap=""><!---->
Maybe I'm missing something, but how does this help?
It may find the first file to copy a little quicker, but finishing the
rsync will take exactly the same time, won't it?
</pre>
</blockquote>
Not at all. If the two are done linearly, then only after the entire
directory tree is scanned will the first transfer *begin*. The total
transfer time will be tree scan time + transfer time for older rsyncs,
but the two overlap for newer transfers. How much time exactly that
would save really depends on how much the second time is (i.e. - how
much data you need to actually transfer).<br>
<blockquote cite="mid:20100412084016.GA27687@fermat.math.technion.ac.il"
type="cite">
<pre wrap="">Also, if nothing has changed, it will take it exactly the same time to
figure this out, won't it?
</pre>
</blockquote>
Yes. You might still save some time, but this, definitely, is the
minimal advantage that newer rsyncs have over older ones.<br>
<blockquote cite="mid:20100412084016.GA27687@fermat.math.technion.ac.il"
type="cite">
<pre wrap="">
I'm not sure what his problem is, though. Is it the fact that the remote
rsync takes a very long time to walk the huge directory tree, or the fact
that sending the whole list over the network is slow?
</pre>
</blockquote>
>From my experience, it's mostly the former.<br>
<blockquote cite="mid:20100412084016.GA27687@fermat.math.technion.ac.il"
type="cite">
<pre wrap="">If it's the first problem, then maybe switching to a different filesystem,
</pre>
</blockquote>
At the time, we tested ext3, jfs and xfs, and found no significant
differences between them. It was not, however, a scientific test.<br>
<blockquote cite="mid:20100412084016.GA27687@fermat.math.technion.ac.il"
type="cite">
<pre wrap="">or reorganizing your directory structure (e.g., not to have more than a few
hundred files per directory) will help.
</pre>
</blockquote>
That is likely to actually help (<plug>and is why rsyncrypto has
the --ne-nesting option when encrypting file names</plug>), but
is not always a viable option.<br>
<blockquote cite="mid:20100412084016.GA27687@fermat.math.technion.ac.il"
type="cite">
<pre wrap="">If it's the second problem, then maybe rsync improvements are due - i.e., to
use rsync's delta protocol not only on the individual files, but also on the
file list.
</pre>
</blockquote>
It's not the second, typically.<br>
<br>
Shachar<br>
<pre class="moz-signature" cols="72">--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
<a class="moz-txt-link-freetext" href="http://www.lingnu.com">http://www.lingnu.com</a>
</pre>
</body>
</html>