<div dir="ltr">
<br><div class="gmail_quote">On Mon, Apr 12, 2010 at 5:02 PM, Nadav Har'El <span dir="ltr"><<a href="mailto:nyh@math.technion.ac.il">nyh@math.technion.ac.il</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
On Mon, Apr 12, 2010, Tom Rosenfeld wrote about "Re: faster rsync of huge directories":<br>
<div class="im">> I realized that in my case I did not really need rsync since it is a local<br>
> disk to disk copy. I could have used a tar and pipe, but I like cpio:<br>
><br>
<br>
</div>Is this quicker?<br></blockquote><div> </div><div>I can't tell, because it is still running, and will be for a few days, but at least it has started copying instead of just building an index.<br> <br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
If it is, then the reason of rsync's extreme slowness which you described<br>
was *not* the filesystem speed. It has to be something else. Maybe rsync<br>
simply uses tons of memory, and starts thrashing? (but this is just a guess,<br>
I didn't look at it code). If this is the case then the copy-while-building-<br>
the-list that Shachar described might indeed be a big win.<br>
<div class="im"><br>
> find $FROMDIR -depth -print |cpio -pdma $TODIR<br>
><br>
> By default cpio also will not overwrite files if the source is not newer.<br>
<br>
</div>I recommend you use the "-print0" option to find instead of -print, and<br>
add the -0 option to cpio. These are GNU extensions to find and cpu (and<br>
a bunch of other commands as well) that uses nulls, instead of newlines,<br>
to separate the file names. This allows newline characters in filenames<br>
(these aren't common, but nevertheless are legal...).<br>
<br>
By the way, while "cpio -p" is indeed a good historic tool, nowadays there<br>
is little reason to use it, because GNU's "cp" make it easier to do almost<br>
everything that cpio -p did: The "-a" option to cp is recursive and copies<br>
links, modes, timestamps and so on, and the "-u" option will only copy if the<br>
source is newer than the destination (or the destination is missing). So,<br>
<br>
cp -au $FROMDIR $TODIR<br>
<br>
is shorter and easier to remember than find | cpio -p. But please note I<br>
didn't test this command, so don't use it on your important data without<br>
thinking first!<br>
<div class="im"><br></div></blockquote><div>Thanks for the tip Nadav (and everyone else.)<br><br>While we are on the
topic, I use cpio because I am also "historic" :-) In the past I had to
do similar copies on diff versions of *NIX (even before rsync was
invented!)<br>
and after much testing of issues of hard links, sym links, timestamps,
etc I found cpio to be the most portable tool. I guess when I get a
chance I will test 'cp -au'<br><br>Thanks,<br>-tom <br></div><br></div></div>