Minimal reproducible example

I have created a minimal reproducible example that can be cloned and easily ran on Github for testing: https://github.com/VirxEC/python-no-run

If you don't want to go to GitHub then the code snippets will be at the bottom of this post.

The issue, described as quickly as possible

When launching Python from Rust on Windows, (Linux works fine,) reading from stdin prevents the Python subprocess from launching more Python subprocesses. Adding a quick sleep(1) lets the subprocess start and it works fine, but if the subprocess takes more than 1 second to start then I'm out of luck. If it takes less than 1 second, then I'm waiting longer than I have to. So, using sleep is the quick patch I've implemented for now but I need a proper solution.

The issue, described in more detail

When and a bit of why

This issue only happens when Python is launched from Rust on Windows only and if you run Python normally then everything works as expected. However, my app isn't a Python app, it's a Rust GUI using Tauri with a Python sub-component because there's a big library that's Python-only and hasn't yet been ported to Rust... and might never get ported due to its large size.

This also means that I can't take other unrelated commands from the main Rust process while it's spawning the subprocesses because it's just not reading stdin and looking for them. In my real program, I start a new "thread" from threading and so stdin can theoretically be read while subprocesses get started on the side. Now I must not let stdin be read while those processes are being spawned, which just adds extra complexity that shouldn't be needed.

What should happen vs what does happen

The issue is that "Hello World" doesn't get spammed because the subprocess spawns but doesn't start executing. I've been at this, and I'm at a loss as to wth is happening. I know it's related to stdin, but my program uses stdin and stdout to communicate between the main Rust process and the Python process.

Somehow reading from stdin on the main thread blocks the subprocess from starting all the way. I know this because if you spawn the subprocess with shell=True then after the loop exits "Hello World" gets spammed.

Also, if I add sleep(1) right after spawning the subprocess, then everything works! Yay! Except this is FAR from ideal. It's like a bandaid that's not quite big enough for the wound, but at least it covers some of it. I've implemented this solution temporarily into my program (specifically sleep(3) for every subprocess spawned) but it's possible for many dozens of processes to need to be spawned which ends up taking much too long. Additionally, it's still possible for a process to just take longer than 3 seconds to start, and I'd be out of luck if that happened.

The borked code (Windows only!)

I've removed import statements etc because they shouldn't be terribly relevant. They can still be found in the GitHub repo that houses the whole minimal reproducible example if you're interested.

main.rs

// Setup command
let mut command = Command::new("python");
command.args(["-u", "-c", "from main import start; start()"]);
command.current_dir(std::env::current_dir()?.join("python"));

// Pipe stdin so we can issue commands to tell Python what we want it to do
command.stdin(Stdio::piped());

// Spawn the process & take ownership of stdin
let mut child = command.spawn()?;
let mut stdin = child.stdin.take().unwrap();

// Give a second to make 100% sure Python has started
sleep(Duration::from_secs(1));

// Issue start command to Python, wait 5 seconds then tell it to stop

println!("Writing start");
stdin.write_all(b"start | \n")?;

sleep(Duration::from_secs(5));

println!("Writing stop");
stdin.write_all(b"stop | \n")?;

// Make sure the process has exited before we exit
child.wait()?;

main.py

def start():
    procs = []
    while True:
        print("Listening for new command...")
        command = sys.stdin.readline()
        params = command.split(" | ")
        print(f"Got command: {command}")

        if params[0] == "stop":
            print("Stopping...")
            # Break from loop
            break
        elif params[0] == "start":
            print("Starting...")
            # Spawn the subprocess
            proc = subprocess.Popen([sys.executable, "say-hi.py"])

            # Add the subprocess to the list so we can kill it later
            procs.append(proc)

    print("Shutting down")
    # Terminate processes
    for proc in procs:
        proc.terminate()

say-hi.py

while True:
    print(f"Hello world!", flush=True)
    sleep(1)

My running process

This is similar to the instructions found in README.md in the GitHub repo.

In the root folder

• Run cargo r and see that nothing happens
• In python/main.py change Popen([sys.executable, "say-hi.py"]) to Popen([sys.executable, "say-hi.py"], shell=True)
• Run cargo r and see that "Hello world" gets spammed when the stop command is issued and never stops

In the python folder

• shell can be True or False, it doesn't matter
• Run python -c "from main import start; start()" to start the process
• Enter start | exactly and see "Hello world" gets spammed
• Enter stop | exactly and see "Hello world" stop being spammed, processes all exit successfully

CodePudding user response：

subprocess.Popen with default parameters requests a handle to stdout and stdin from the operating system on windows ... which seems to get stuck or fail for some reason.

anyway passing the parent's stdout or a PIPE seems to fix it on windows, simply replacing

proc = subprocess.Popen([sys.executable, "say-hi.py"])

with

proc = subprocess.Popen([sys.executable, "say-hi.py"], stdin=subprocess.PIPE, stdout=sys.stdout, stderr=sys.stderr)

which simply passes the parent's stdout file ids to the child to use.