Cut a list in 3 parts to speed up Async function-CodePudding

i have a linq expression which does the job:

var y = docs.ToList()
var res = y.Select(async b => await loadAgence(b.url))
           .Select(x => x.Result)
    :
    :
private async Task<BsonDocument? doc, int status, bool iSSucess) loadAgence(string url)
{
    var (docs, status, isSuccess) = await Helper.API_GetDocument(url);
    return (isSuccess ? docs : null, status, isSuccess)
}

this job takes 3 minutes to execute 900 loops (900 Call API)

i would cut the collection in 3 parts, to divise by 3 the time lapse

var res1 = y.Skip(0).Take(300).Select(async b => await loadAgence(b.url))
            .Select(x => x.Result)
var res2 = y.Skip(300).Take(300)Select(async b => await loadAgence(b.url))
            .Select(x => x.Result)
var res3 = y.Skip(600).Take(300)Select(async b => await loadAgence(b.url))
            .Select(x => x.Result)

but of course each part is blocked, how i could resolve this problem?

i have tried to use WhenAll with a list collection (res1 to res3) but its waiting IEnumerable<Task> and i have List<IEnumerable<Task<BsonDocument? doc, int status, bool iSuccess)>>> so it refuses the cast..

is it possible to do that?

CodePudding user response：

but of course each part is blocked, how i could resolve this problem?

Because you use .Select(x => x.Result) for each part of Task list which means that the code will wait for each connection as same as synchronizing code.

If you want to use asynchronous code we might get Task or Task of List for this example we will get IEnumerable<Task(BsonDocument? doc, int status, bool iSSucess)>> from res1, res2,res3

var res1 = y.Skip(0).Take(300).Select(async b => await loadAgence(b.url));
var res2 = y.Skip(300).Take(300).Select(async b => await loadAgence(b.url));
var res3 = y.Skip(600).Take(300).Select(async b => await loadAgence(b.url));

Unfortunately Task.WhenAll might not support multiple IEnumerable<Task> array parameters, so we can try to use List<Task<(BsonDocument? doc, int status, bool iSSucess)>> to append all of list in that then Task.WhenAll

var tasks = new List<Task<(BsonDocument? doc, int status, bool iSSucess)>>();   
tasks.AddRange(res1);   
tasks.AddRange(res2);   
tasks.AddRange(res3);  

var result = await Task.WhenAll(tasks.ToArray());