Ooookay, so this has been a thorn in my side for a good week now.
First off, this is for Framework 4.8 (beyond my control).
I make a list of tasks, using Task.Run and a lambda to call an async function that calls a 3rd party API and returns a: Task<List<Dictionary<string, object>>>
Number of tasks is 75.
The API has pagination requirements, so the calling function takes start and end parameters, based on the iterator of the loop that builds the task list.
So far all straight forward.
Also straight forward is that after Task.WaitAll I get the wrong results, because closures close over variables, not over values. The iterator is at max long before the tasks have spun off, and thus the vast majority end up with the wrong pagination.
**** However, they do all run to completion.****
Now, whenever I try to solve the closure issue, I end up with a consistent result where only some of the tasks run to completion and the rest all fail. I have tried:
- Assigning the iterator to a new variable in the lambda
- Using
Task.Factory.StartNewand passing the iterator in as a state object - Variations of the above, but with the start and end calculations moved to the calling function
In all cases the result remains the same: anywhere from a handful to a third of the tasks complete and the rest all fail, usually the former.
The AggregateException collection offers little to no insight as far as I can understand.
The general exception is the usual "One or more errors have occurred" and the inner exceptions hints at a movenext failure(?). Stack traces are minimal one-liners.
What could be going on here?
Thanks for the help.
-Willem
The calling function
public async Task<List<Dictionary<string, object>>> ExecuteAsyncJSON(
string requestSource = "fields", string searchCount = "", Object iteration = null)
{
string url = requestSource;
// Add possible parameters.
if (searchCount != "" && iteration != null)
{
// divide count by 500 remainder for pagination
int quotient = (Int32.Parse(searchCount) / 500);
int remainder = Int32.Parse(searchCount) % 500;
int i = Convert.ToInt32(iteration);
int add = i == quotient ? remainder : 500;
int start = i * 500;
int end = (i * 500) add;
if (url.Contains("?"))
{
url = url "&start=" start.ToString() "&end=" end.ToString();
}
else
{
url = url "?start=" start.ToString() "&end=" end.ToString();
}
}
try
{
// make the call and get the response
using (HttpResponseMessage response = await someclient.GetAsync(url))
// this calls the API through an HttpClient
{
if (response.IsSuccessStatusCode)
{
var serializer = new JavaScriptSerializer();
var responseBodyAsText = await response.Content.ReadAsStringAsync();
var returnData = new List<Dictionary<string, object>>();
returnData = serializer
.Deserialize<List<Dictionary<string, object>>>(responseBodyAsText);
return returnData;
}
else
{
throw new Exception(response.ReasonPhrase);
}
}
}
catch
{
throw new Exception();
}
}
The main stuff that tries to get this all done
var requestSource = "searches/" searchId "/members";
var searchCount = 37301;
int quotient = (Int32.Parse(searchCount) / 500);
var tasks = new List<Task<List<Dictionary<string, object>>>>();
// Put the task list together.
// Using the StartNew with state object to pass the iterator in.
for (var i = 0; i <= quotient; i )
{
var task = Task.Factory.StartNew((iteration) =>
{
var returnData = ExecuteAsyncJSON(requestSource, searchCount, iteration);
return returnData;
}, i)
.Unwrap();
tasks.Add(task);
}
try
{
Task.WaitAll(tasks.ToArray());
}
catch (AggregateException ae)
{
//throw ae;
}
List<Dictionary<string, object>> completedResults
= new List<Dictionary<string, object>>();
List<Task<List<Dictionary<string, object>>>> faultedResults
= new List<Task<List<Dictionary<string, object>>>>();
foreach (Task<List<Dictionary<string, object>>> listTask in tasks)
{
if (listTask.Status == TaskStatus.RanToCompletion)
completedResults.AddRange(listTask.Result);
if (listTask.Status == TaskStatus.Faulted)
faultedResults.Add(listTask);
}
Debug Data
The WaitAll AggregateException is:
One or more errors occurred
Exception of type 'System.Exception' was thrown.
at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout)
at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
at Smu.Bbec.Emma.API.EmmaAPI.<GetMembersMatchingSearch>d__22.MoveNext() in [reference to Task.WaitAll(tasks.ToArray());]
Out of 76 calls, 4 completed just fine and then 72 fail with the exact same inner exception:
System.Exception: Exception of type 'System.Exception' was thrown.
at Smu.Bbec.Emma.API.EmmaAPI.<ExecuteAsyncJSON>d__8.MoveNext() in [reference to code that makes the actual call to the API]
I set some breakpoints and this is the result:
A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Smu.Bbec.Emma.API.EmmaAPI.<ExecuteAsyncJSON>d__8.MoveNext() in f:\\TestApps\\Smu.Bbec.Emma\\Smu.Bbec.Emma.API\\EmmaAPI.cs:line 100"
No inner exception.
And this variety;
An error occurred while sending the request.
Inner exception: The request was aborted: The request was canceled (System.Exception {System.Net.WebException})
This is the full debug data I have on the cancelation (sensitive data removed):
$exception {"A task was canceled."} System.Threading.Tasks.TaskCanceledException
this {**********.EmmaAPI} **********.EmmaAPI
_accountId "********" string
_authorization "MTg4NzblahblahblahblahmM6MDNmYzU1blahblahblahblahjM=" string
_baseUri {https://api.e2ma.net/********/} System.Uri
_baseUrl "https://api.e2ma.net/********/" string
_publicKey "********" string
_secretKey "********" string
emmaClient {System.Net.Http.HttpClient} System.Net.Http.HttpClient
BaseAddress {https://api.e2ma.net/********/} System.Uri
DefaultRequestHeaders {Accept: application/json Authorization: Basic MblahblahblahblahTFjYmblahblahblahblahM4M2IyMjM=} System.Net.Http.Headers.HttpRequestHeaders
MaxResponseContentBufferSize 2147483647 long
Timeout {00:01:40} System.TimeSpan
baseAddress {https://api.e2ma.net/********/} System.Uri
defaultRequestHeaders {Accept: application/json Authorization: Basic MTgblahblahblahblah4Y2blahblahblahblahNmYzblahblahblahblahjM=} System.Net.Http.Headers.HttpRequestHeaders
disposeHandler true bool
disposed (System.Net.Http.HttpMessageInvoker) false bool
disposed false bool
handler {System.Net.Http.HttpClientHandler} System.Net.Http.HttpMessageHandler {System.Net.Http.HttpClientHandler}
maxResponseContentBufferSize 2147483647 long
operationStarted true bool
pendingRequestsCts {System.Threading.CancellationTokenSource} System.Threading.CancellationTokenSource
timeout {00:01:40} System.TimeSpan
Static members
requestSource "searches/17297143/members" string
searchCount "37810" string
iteration 67 object {int}
url "searches/17297143/members?start=33500&end=34000" string
CodePudding user response：
I found the answer to my conundrum.
It turned out to be the same issue as posted here:
c# a task was canceled at the await call
The solution for me was to add .Timeout = TimeSpan.FromMinutes(30) to my HttpClient.
Thanks to Theodor Zoulias, Stephen Cleary, and everyone else who helped me out here.
Cheers!
CodePudding user response：
Replace the loop with LINQ extension methods, toss the superfluous task/unwrap and see if that helps you.
var requestSource = ...;
var searchCount = ...;
var quotient = ...;
var tasks = Enumerable.Range(0, quotient)
.Select(i => ExecuteAsyncJSON(requestSource, searchCount, i))
.ToArray();
Task.WaitAll(tasks);