C# Process.Start() within Thread shortly freezes Main-Thread-CodePudding

We are currently evaluating some slowish performance on an application, and found a strange behaviour.

While the code runs perfectly fine on Windows 7 / Server 2008 R2 / Server 2012 R2, starting with Server 2019 / Windows 10/11, we can notice a strange stuck on the Main-Thread.

All Systems using .net Framework 4.8:

The main-thread is iterating over a list of tasks, where foreach task a thread is started, hardly simplified like this:

foreach (CustomTaskObject task in this.Tasks){
    task.ThreadObject.start();
}

Each Task-Object wraps basically either a cmd-script or a powershell script, so the tasks objective is to run that script, transscript the output, wait for finish and log.

The ThreadObject is basically created like this:

public class CustomTaskObject

     public Thread ThreadObject;

     public CustomTaskObject(){
          this.ThreadObject = new Thread(new ThreadStart(new Action(() => {
                Process p = new Process();
                p.StartInfo.FileName = "powershell.exe"; //cmd.exe /C if a batch-script.
                p.StartInfo.Arguments = "-ExecutionPolicy Bypass -f "   this.Folder   "\\"   this.File;

               p.Start();

               p.WaitForExit();
          }));
     }

All the stuff for redirecting output / reading / logging has been removed, it has no impact, also it doesn't matter if we use ShellExecute True or false.

What happens now is:

For cmd-calls, everything runs nice and smooth. If Tasks are defined to be executable asynchronous, the main thread can create 50 threads in no time, Main-Thread stays active / responsive.
For powershell-calls: As soon as p.Start() is called, the THREAD is stucking about 2-3 Seconds, if the file is located on a network share. This is no big surprise, dns resolution, smb paket stuff, building connections - takes some time, all good.

However, what is really strange: The Main-Thread is stuck at calling task.ThreadObject.start(); as well - Until the Process in the thread is started. This makes the main thread stuck while iterating through the job-objects, and you can see that one is started about every 3 seconds.

For testing purpose we added a Thread.Sleep(1000) before p.Start() - and guess what? The mainthread was able to kick off 30 these threads, before one of them was causing a stuck...

public class CustomTaskObject

     public Thread ThreadObject;

     public CustomTaskObject(){
          this.ThreadObject = new Thread(new ThreadStart(new Action(() => {
                Process p = new Process();
                p.StartInfo.FileName = "powershell.exe"; //cmd.exe if a ps-script.
                p.StartInfo.Arguments = "-ExecutionPolicy Bypass -f "   this.Folder   "\\"   this.File;

               System.Threading.Thread.Sleep(1000);
               p.Start();

               p.WaitForExit();
          }));
     }

So, what is happening here? I can only assume, that calling process.Start() is in someway offloaded to the Main-Thread, even if started from within a thread? Hence, If we delay that by 1000ms, the main thread can finish iteration first, then get stuck (invisible) to start processes.

Why does this only happen for the mentioned operating systems?
Why does it run flawless for cmd.exe targeting network files in the very same way?

I can even reproduce this with the following console-Application:

From the output, you can see, where the first time p.Start() is reached - leading to stucks of about 1 second here, summing up, as more p.starts() are reached by their threads.

class Program
{
    static Thread[] threads = new Thread[30];

    static void Main(string[] args)
    {
        for (int i = 0; i < 30; i  )
        {
            threads[i] = new Thread(new ThreadStart(new Action(() =>
            {
                Process p = new Process();
                 p.StartInfo.FileName = "powershell.exe";
                p.StartInfo.Arguments = @"-ExecutionPolicy Bypass -f \\server\share\ping.ps1";
                p.Start();
                p.WaitForExit();

            })));
        }

        for (int i = 0; i < 30; i  ) 
        {
            Console.WriteLine(DateTime.Now.ToLongTimeString()   " Starting thread "   i);
            threads[i].Start();
            Console.WriteLine(DateTime.Now.ToLongTimeString()   " Thread started "   i);
        }

        Console.ReadKey();
    }
}

Output:

17:30:20 Starting thread 0
17:30:20 Thread started 0
17:30:20 Starting thread 1
17:30:20 Thread started 1
17:30:20 Starting thread 2
17:30:20 Thread started 2
17:30:20 Starting thread 3
17:30:20 Thread started 3
17:30:20 Starting thread 4
17:30:20 Thread started 4
17:30:20 Starting thread 5
17:30:20 Thread started 5
17:30:20 Starting thread 6
17:30:20 Thread started 6
17:30:20 Starting thread 7
17:30:20 Thread started 7
17:30:20 Starting thread 8
17:30:20 Thread started 8
17:30:20 Starting thread 9
17:30:20 Thread started 9
17:30:20 Starting thread 10
17:30:20 Thread started 10
17:30:20 Starting thread 11
17:30:25 Thread started 11
17:30:25 Starting thread 12
17:30:29 Thread started 12
17:30:29 Starting thread 13
17:30:31 Thread started 13
17:30:31 Starting thread 14
17:30:33 Thread started 14
17:30:33 Starting thread 15
17:30:33 Thread started 15
17:30:33 Starting thread 16
17:30:36 Thread started 16
17:30:36 Starting thread 17
17:30:37 Thread started 17
17:30:37 Starting thread 18
17:30:40 Thread started 18
17:30:40 Starting thread 19
17:30:41 Thread started 19
17:30:41 Starting thread 20
17:30:42 Thread started 20
17:30:42 Starting thread 21
17:30:44 Thread started 21
17:30:44 Starting thread 22
17:30:45 Thread started 22
17:30:45 Starting thread 23
17:30:46 Thread started 23
17:30:46 Starting thread 24
17:30:47 Thread started 24
17:30:47 Starting thread 25
17:30:48 Thread started 25
17:30:48 Starting thread 26
17:30:49 Thread started 26
17:30:49 Starting thread 27
17:30:50 Thread started 27
17:30:50 Starting thread 28
17:30:52 Thread started 28
17:30:52 Starting thread 29
17:30:53 Thread started 29

Sure, 30 threads cannot be executed at the same time, unless you have a 30 core cpu - but how can it "stuck" the Main-Thread on Windows Server 2019 / Windows 10 & 11?

Updated the test-script to access a network share. After 11 threads, the first p.start() is actually reached, leading to significant stucks on the main-thread.

Running on a 20 core, I can assume the main-thread didnt get suspended. It seems the whole stuck is exactly in place, until powershell has resolved and loaded that file. Then the main thread continues, while the script is running in its process as it is supposed to be.

Funny observation: While the Main-Thread is stucking, even Visual Studios profiler is stucking... (But it also happens when running the app native, without debugger attached)

CodePudding user response：

I've found a similiar issue here: Process.Start() hangs when running on a background thread

The thread is deadlocked on the Console.InputEncoding property getter. 
Which is used by the Process class to figure out what encoding needs to be 
used to translate the redirected output of the process into strings.

and here Process spawned through Process.Start in .NET hangs the thread

As a workaround you may use the start command. The following code launches 
a hidden shell window which "starts" the url:

So, I tested this, and it is interesting. Using

 Process p = new Process();
 p.StartInfo.FileName = "cmd.exe";
 p.StartInfo.Arguments = @"/c start powershell.exe -ExecutionPolicy Bypass -f \\server\share\ping.ps1";
 p.Start();

runs smooth upto exactly 20 threads (got a 20 core cpu) - then the main-thread stucks until a first child completes. But that is a more expected behavior, and I'm usually not launching more threads than cores-1 anyway.

So, putting both things together, it has to do with the applications std output stream, and that each started process interacts with that as well, causing even the mainthread to stuck during process.start(). (Which cmd /C start is avoiding)

I'll do some more testing later if this can be avoided in "nicer" ways, but I fear that as long as i want the output stream of that process beeing catched, there is little room... Maybe just something hacky like piping to a tempfile and reading afterwards.

CodePudding user response：

Turns out - a proper solution is sometimes better than a workaround.

There is indeed a connection between the process start in the thread and a stucking main-thread - but since this didn't happen on Windows 7 machines and not for "cmd"-starts, I checked for the reason of the "slow powershell" start at all.

It has been Kasperky, who was delaying each call to powershell.exe by 2-4 seconds. Shutting down the clown, and immediately p.start() returns in no time, which in turn did then no longer cause any noticable stucks on the main-thread.

Guess this is part of the Live-Threat-Defence, where Kaspersky was "freezing" the called application (powershell) as well as the caller-stack (process.start, main-thread, upto visual studio), until it figured that everything looks serious.