Our ASP NET software has this specific condition where Parallel.ForEach is applied. It was working fine, but we recently noticed that it stopped using all available cores for these specific heavy operations where parallelism is applied.
This is the code we use:
Parallel.ForEach(var1, new ParallelOptions
{ MaxDegreeOfParallelism = Environment.ProcessorCount - 1 }, var2 => ...
What we used to see at Task Manager was a high usage (like 90%) of our 32x cores server while running these specific heavy operations. But now it keeps itself using only one core (~5% only). What used to take 20-40 min now is taking 5-6 hours!
While on our computer (running locally with Visual Studio) we do see that all cores are working as expected (~80-90% usage).
Details: Server has 2x CPUs, x64, Windows Server 2012 bare metal (no VM). There are no hardware problems. Software runs at ASP NET 4.6.2. MaxDegreeOfParallelism correctly returns 32. IIS Manager / Advanced Settings: CPU limit = 900. Maximum Worker Process seems to have no difference. Database doesn't seem to be the problem because all data are previously stored at memory (lists variables at the code).
As the code used to work fine at this same server, and currently works fine on our localhost, we suspect that something changed at the server side.
We suspect that a Microsoft update may be the one causing this, although we didn't found anything so far. These are the latest 2022 updates that we suspect:
2022-05 Security and Quality Rollup for .NET Framework 3.5, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 for Windows Server 2012 for x64 (KB5013871). Installation date: 25/05/2022 09:43
2022-04 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 for Windows Server 2012 for x64 (KB5012330). Installation date: 20/04/2022 15:53
2022-02 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 for Windows Server 2012 for x64 (KB5010582). Installation date: 11/02/2022 20:27
2022-01 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8 for Windows Server 2012 for x64 (KB5009720). Installation date: 12/01/2022 21:48
Does anyone know something about this issue? Any ideas on how to troubleshoot this?
CodePudding user response：
Any ideas on how to troubleshoot this?
You could try running this experiment on your server, as a standalone Console application:
ParallelOptions options = new()
{
MaxDegreeOfParallelism = Environment.ProcessorCount
};
int concurrency = 0;
Parallel.ForEach(Enumerable.Range(1, 100), options, item =>
{
var current = Interlocked.Increment(ref concurrency);
try
{
Console.WriteLine($"Processing {item}, Concurrency: {current}");
for (int i = 0; i < 1_000_000_000; i ) { }
}
finally { Interlocked.Decrement(ref concurrency); }
});
The expected behavior is to see the CPU maxing out (100%), and the reported concurrency reaching the value 32.
CodePudding user response：
It's by design that Parallel.ForEach may use fewer threads than requested to achieve better performance. According to MSDN [link]:
By default, the Parallel.ForEach and Parallel.For methods can use a variable number of tasks. That's why, for example, the ParallelOptions class has a MaxDegreeOfParallelism property instead of a "MinDegreeOfParallelism" property. The idea is that the system can use fewer threads than requested to process a loop.
The .NET thread pool adapts dynamically to changing workloads by allowing the number of worker threads for parallel tasks to change over time. At run time, the system observes whether increasing the number of threads improves or degrades overall throughput and adjusts the number of worker threads accordingly.