Job seems to run as Fire and Forget. Database activity continues for up to 10 minutes after success-CodePudding

I am debugging this hangfire service job call and can't figure out why hangfire is reporting a completed state while the stored procedures being called are continuing to run for up to ~10 minutes later. The parallel loop is running a stored procedure against up to 200 databases.

The first image below shows the hangfire status (success in ~40 seconds), meaning the job function completed. The second image shows the Stored Procedures issued form the loop still running for many minutes later. As far as I know, hangfire will report success when the job function completes, report timeout after a max wait, or report error is an exception occurs. I don't see how this function could be exiting in 6 seconds if all tasks are awaited.

Hangfire reports success in 40 seconds

Long Running Query Detection shows queries from the job still running and falling in and out of blocking state many minutes later. The commands starting with --===================== represent the SP running against many different databases.

Code I have started debugging ->

public async Task PerformServiceWork()
{
    try
    {
        var adminDatabases = await _DatabaseService.GetActiveDatabasesAsync();
               
        Parallel.ForEach(adminDatabases,
            new ParallelOptions { MaxDegreeOfParallelism = Convert.ToInt32(Math.Ceiling((Environment.ProcessorCount * 0.75) * 2.0)) },  // Suggested: MaxDegreeOfParallelism to use a maximum of 75% resources of the system, each processor contains two cores
            async clientDatabase =>
            {
                double elasped = 0.00;
                try
                {
                            
                    var connectionString = _ASettingsHttpApiProvider.GetConnectionString(clientDatabase.SystemName);  

                    if (!string.IsNullOrEmpty(connectionString))
                    {
                        LogHelper.LogInfo(clientDatabase.SystemName, $"Starting at {DateTime.Now:HH:mm:ss.fff tt}");

                        elasped = await UpdateLastDetectedChangeDateAndAlertFlag(clientDatabase, connectionString);

                        LogHelper.LogInfo(clientDatabase.SystemName, $"Ended at {DateTime.Now:HH:mm:ss.fff tt}  Total: {elasped} seconds");
                    }
                }
                catch (Exception ex)
                {
                    LogHelper.LogException(LogLevel.Error, MethodBase.GetCurrentMethod(), clientDatabase.SystemName, ex);
                }

            });

    }
    catch (Exception ex)
    {
        LogHelper.LogException(LogLevel.Error, MethodBase.GetCurrentMethod(), "Lease Last Detected Change Job", ex);
    }
}


private async Task<double> UpdateLastDetectedChangeDateAndAlertFlag(tblDatabase clientDatabase, string connectionString)
{
    _ProcessTimer.Restart();

    try
    {
        using var connection = new SqlConnection().AsReducedPoolConnection(clientDatabase, connectionString);
        await connection.ExecuteAsync("AStoredProcedureThatTakesAbout3To5SecondsToRun", commandType: CommandType.StoredProcedure, commandTimeout: _CommandTimeOutInSeconds);
    }
    catch (Exception ex)
    {
        LogHelper.LogException(LogLevel.Error, MethodBase.GetCurrentMethod(), clientDatabase.SystemName, ex);
    }

    _ProcessTimer.Stop();

    return _ProcessTimer.Elapsed.TotalSeconds;
}

CodePudding user response：

My thoughts are that Parallel.ForEach does not await for the async lambda so PerformServiceWork ends before each lambda is still running.

Try to use instead Task.WhenAll and await on it.

As I recall Parallel.ForEach is more related to Threading while Tasks are an abstraction over Threads.