Excluding a list of files from tar with spaces in filenames

[Amos Shapira]

On 6 February 2010 00:24, Dotan Cohen <dotancohen at gmail.com> wrote:
>> OK, fair enough, but why do you need the 'dd' at the end?:
>>
>
> Because I know nothing about shell scripting and I am learning by
> google, trial and error.

No worries. I still sometimes get a kick from showing off my
shell/scripting/programming prowess :)

>
>
>> tar -zcvf - --exclude-from $EXCLUDES * 2>STDERR | openssl des3 -salt
>> -k $1 > $(hostname)-$(date +%Y%m%d).tbz
>>
>> should output tar's stderr to STDERR and tar's output will be piped to openssl.
>
> I am still not clear about STDERR. I need the output (the output from
> the v flag) in a variable.

Since you instruct tar to output the tar file itself to its standard
output, tar does the right thing and outputs the list of files names
to stderr. Here is a simple experiment:

$ touch x
$ tar zcvf - x 2>tar.err | cat > x.tar
$ cat tar.err
x
$ rm x.tar
$ tar zcvf x.tar x 2>tar.err
x
$ cat tar.err
$

i.e. if tar's output goes to stdout then the output of "v" will go to
stderr (so it doesn't mix with tar's output file), otherwise it goes
to stdout.

That's why to capture the output of tar's "v" flag when piping tar's
output via a pipe you need to capture stderr, that's done using the
"2>" redirection ("2>" means "redirect file descriptor number 2
(standard error)", look under "REDIRECTIONS" in "man bash").

>> Notice how I moved the "*" to the end of tar's command line, btw.
>>
>
> What is the advantage of this?

Most unix commands (with some small exceptions) stop processing
command line flags and treat the rest of the list of their command
line args as input files to process once they find a none-flag
argument. Once the "*" gets expanded (by the shell, that's why you
have this mess with spaces in file names) into file names and tar sees
them, it will treat the --exclude-from that comes after them as yet
another file name, not a command line flag.

>
>
>> If you want to redirect the entire command line output to STDERR use
>> parenthesis around it:
>>
>> (tar -zcvf - --exclude-from $EXCLUDES * | openssl des3 -salt -k $1 >
>> $(hostname)-$(date +%Y%m%d).tbz) 2>STDERR
>>
>
> How about if I need it in the variable COPIED instead of STDERR?

First - just to be clear on terminology, STDERR in the command above
is an example of a file name, not a variable.

Second - to achieve what you want to can either:
1. Surround the entire pipeline in parenthesis and redirect stderr to
stdout, which will be picked up by back-ticks:

COPIED=`(tar -zcvf - --exclude-from $EXCLUDES * | openssl des3 -salt
-k $1 > $(hostname)-$(date +%Y%m%d).tbz) 2>&1`

(note that these a back-ticks (the key left of the "1" key), not plain
simple quotes).

2. maybe a bit clearer to read is to keep the output in the STDERR
file then read it into a variable in a separate command:

(tar -zcvf - --exclude-from $EXCLUDES * | openssl des3 -salt -k $1 >
$(hostname)-$(date +%Y%m%d).tbz) 2>STDERR
COPIED=$(< STDERR)

(or to avoid bash-ism syntax you can replace the second by:
COPIED=`cat STDERR`

Hope this answers your question.

Cheers,

--Amos