faster rsync of huge directories

[Nadav Har'El]

On Mon, Apr 12, 2010, Tom Rosenfeld wrote about "Re: faster rsync of huge directories":
> I realized that in my case I did not really need rsync since it is a local
> disk to disk copy. I could have used a tar and pipe, but I like cpio:
> 

Is this quicker?
If it is, then the reason of rsync's extreme slowness which you described
was *not* the filesystem speed. It has to be something else. Maybe rsync
simply uses tons of memory, and starts thrashing? (but this is just a guess,
I didn't look at it code). If this is the case then the copy-while-building-
the-list that Shachar described might indeed be a big win.

>   find $FROMDIR -depth -print |cpio -pdma  $TODIR
> 
> By default cpio also will not overwrite files if the source is not newer.

I recommend you use the "-print0" option to find instead of -print, and
add the -0 option to cpio. These are GNU extensions to find and cpu (and
a bunch of other commands as well) that uses nulls, instead of newlines,
to separate the file names. This allows newline characters in filenames
(these aren't common, but nevertheless are legal...).

By the way, while "cpio -p" is indeed a good historic tool, nowadays there
is little reason to use it, because GNU's "cp" make it easier to do almost
everything that cpio -p did: The "-a" option to cp is recursive and copies
links, modes, timestamps and so on, and the "-u" option will only copy if the
source is newer than the destination (or the destination is missing). So,

	cp -au $FROMDIR $TODIR

is shorter and easier to remember than find | cpio -p. But please note I
didn't test this command, so don't use it on your important data without
thinking first!

-- 
Nadav Har'El                        |       Monday, Apr 12 2010, 28 Nisan 5770
nyh at math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |I have a watch cat! If someone breaks in,
http://nadav.harel.org.il           |she'll watch.