I want redirect output from scp into a file, but only lines that match a pattern. For example, I'm getting a lot of permission denied errors, so I want to log all the files where that is the case. I found that I can redirect all output of scp into a file like this:

source my_scp_script.sh > output.log 2>&1

The script is just a simple call to scp. I'm stuck on how I can match a pattern like "Permission Denied" and only write those lines to the file, not everything since there are thousands of files that are successful.

EDIT: I forgot to clarify that I have tried using grep, but it does not work when doing it like this source my_scp_script.sh | grep Permission > output.log

CodePudding user response：

There is nothing special about scp; you just need to get familiar with shell I/O redirections. (The Linux Documentation Project's Advanced Bash-Scripting Guide is IMO a great resource of information; §20 I/O Redirection contains a nice summary.)

You could use source my_scp_script.sh 1> /tmp/stdout 2> /tmp/stderr to redirect the output written to the file descriptors 1 (aka stdout; note that 1> can also be written as >, which is more commonly used) and 2 (aka stderr) to temporary files. You can then inspect /tmp/std{out,err} and will find that the Permission Denied errors are written to stderr.

A simple pipeline | grep connects stdout with grep's stdin, but as you need to connect stderr, you can use

source my_scp_script.sh 2>&1 | grep "Permission Denied" > output.log

to redirect file descriptor 2 to file descriptor 1 before piping the combined stdout to grep.

With |& being a shortcut for 2>&1 |, you can finally simplify this to:

source my_scp_script.sh |& grep "Permission Denied" > output.log