How can I run ripgrep using subprocess.Popen in Python3 with arguments?-CodePudding

I use Python 3.10.7 and I am trying to get the Python interpreter to run this command:

rg mysearchterm /home/user/stuff

This command, when I run it in bash directly successfully runs ripgrep and searches the directory (recursively) /home/user/stuff for the term mysearchterm. However, I'm trying to do this programmatically with Python's subprocess.Popen() and I am running into issues:

from subprocess import Popen, PIPE

proc1 = Popen(["rg", "term", "/home/user/stuff", "--no-filename"],stdout=PIPE,shell=True)
proc2 = Popen(["wc","-l"],stdin=proc1.stdin,stdout=PIPE,shell=True)

#Note: I've also tried it like below:
proc1 = Popen(f"rg term /home/user/stuff --no-filename",stdout=PIPE,shell=True)
proc2 = Popen("wc -l",stdin=proc1.stdin,stdout=PIPE,shell=True)


result, _ = proc2.communicate()
print(result.decode())

What happens here was bizarre to me; I get an error (from rg itself) which says:

error: The following required arguments were not provided: <PATTERN>

So, using my debugging/tracing skills, I looked at the process chain and I see that the python interpreter itself is performing:

python3          1921496 953810   0 /usr/bin/python3 ./debug_script.py
sh               1921497 1921496   0 /bin/sh -c rg term /home/user/stuff --no-filename
sh               1921498 1921496   0 /bin/sh -c wc -l

So my next thought is just trying to run that manually in bash, leading to the same error. However, in bash, when I run /bin/sh -c "rg term /home/user/stuff --no-filename" with double quotations, the command works in bash but when I try to do this programmatically in Popen() it again doesn't work even when I try to escape them with \. This time, I get errors about unexpected EOF.

CodePudding user response：

As for the behavior when shell=True is specified, the python document tells:

If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

Then your command invocation is equivalent to:

/bin/sh -c "rg" "term" "/home/tshiono/stackoverflow/221215" ...

where no arguments are fed to rg.

You need to pass the command as a string (not a list) or just drop shell=True.