xargs pipe non empty stdin lines to command while preserving double quotes-CodePudding

I'm trying to have a script listen to stdin (so I run it and it doesn't immediately exit) and only execute when stdin is not empty and then pipe the stdin line to another command. Right now I'm using the command from the answer here:

xargs -I {} sh -c 'echo {} | foo'

I want to preserve double quotes from stdin, for that people suggest using -d '\n' but this causes foo to run on empty lines.

I looked into possible GNU Parallel solutions but couldn't find anything.

Here is my stdout:

>xargs -I {} sh -c 'echo {} | foo'


bar
I have executed for 'bar'

"bar"
I have executed for 'bar' //notice the double quotes missing
^C
>xargs -I {} sh -c "echo '{}' | foo"

bar
I have executed for 'bar'

"bar"
I have executed for 'bar' //Same thing, double quotes missing
^C
>xargs -d '\n' -I {} sh -c "echo {} | foo"

i have executed for '' //doesn't ignore empty lines anymore

i have executed for ''
bar
i have executed for 'bar'
"bar"
i have executed for 'bar'

Desired output:

bar
I have executed for 'bar'

"bar"
I have executed for '"bar"'

Running

echo '"bar"' | foo

gets me

I have executed for '"bar"'

CodePudding user response：

If, as your tags suggest, you are running on linux, you have GNU xargs, which supports the -0 option.

Removing empty lines could be accomplished with a simple grep in front. There is also xargs -r which says to not run the command if xargs receives empty input (this too is a GNU extension).

Your attempts are slightly problematic, though; you should pass the arguments as command-line arguments rather than have xargs interpolate them into the sh -c '... {} ...' string literally.

Slightly depending on your requirements, this could even work portably on other platforms:

xargs sh -c 'if [ $# -gt 0 ]; then echo "$@" | foo; fi' _

The _ is just a placeholder; the arguments to sh -c '...' are used to populate $0, $1, $2, etc and so we put in something, anything, to occupy the slot for $0.

CodePudding user response：

GNU Parallel uses this internally:

perl -e 'if(sysread(STDIN,$buf,1)){open($fh,"|-",@ARGV)||die;syswrite($fh,$buf);if($read=sysread(STDIN,$buf,131071)){syswrite($fh,$buf);}while($read=sysread(STDIN,$buf,131072)){syswrite($fh,$buf);}close$fh;exit($?&127?128 ($?&127):1 $?>>8)}' /usr/bin/bash -c 'wc -l'

If you only want a single line try:

seq 3 | parallel --pipe -N1 wc -c
echo "'foo'" | parallel --pipe -N1 --rrs "echo -n i have executed for \"'\";cat;echo \"'\""
echo '"foo"' | parallel --pipe -N1 --rrs "echo -n i have executed for \"'\";cat;echo \"'\""

CodePudding user response：

I want to preserve double quotes from stdin, for that people suggest using -d '\n' but this causes foo to run on empty lines.

xargs performs quote processing by default unless you specify a delimiter via either -d/--delimiter or -0/--null. You must use one of these to avoid xargs removing the quotes you are trying to preserve.

What's more, supposing that you manage to pass the quoted input through xargs unchanged, the shell that xargs launches to run the command will perform its own quote removal, as well as parameter expansion, variable assingments, redirection processing, etc. You can observe the effects of that directly with this variation on your command:

$ xargs -d '\n' -I{} sh -c 'echo {} >>tmp.txt' 
bar
'bar'
$ cat tmp.txt
bar
bar
$

Note that the quotes are removed despite specifying a delimiter to xargs.

It's a bit hard to parse your exact requirements, but it sounds like you just want to filter empty lines out of the standard input to some command. sed can do that pretty easily:

foo() {
  while IFS= read -r line; do
    echo "I have executed for '$line'"
  done
}

$ sed '/\S/!d' | foo


bar

"bar"
A whole line with "quotes" and 'quotes' and metacharacters > < !
I have executed for 'bar'
I have executed for '"bar"'
I have executed for 'A whole line with "quotes" and 'quotes' and metacharacters > < !'
$

Explanation of the sed command: the regex /\S/ matches any non-whitespace character, anywhere on the line. The ! negates the match, and the d deletes lines matching the (negated) pattern -- that is any line that does not contain at least one non-whitespace character.

As you can see in the example run transcript, there is a difference in buffering between your example command and the effect of filtering with sed. It's unclear whether that's important to you.