I need to call one particular endpoint 18000 times using .NET Core 6. I need to log the response from each endpoint call.

The endpoint takes approx. 2 seconds to respond so I want to get all the requests sent quicker than that :)

Pretty sure Task.WhenAll is my friend here?

CodePudding user response：

for (int i = 0; i < 10; i  )
{
    new Thread(async () =>
    {
        //call api here
    }).Start();
}

You can try this. Make sure you test it out with a small sample before you call 18000 times.

CodePudding user response：

You can use an ActionBlock from the Dataflow library for this kind of activity. It will give you control over the level of parallelism and manage all the tasks for you.

https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/dataflow-task-parallel-library

https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.dataflow.actionblock-1?view=net-7.0

The example on how to use this on msdn is perhaps not as clear as it could be, hopefully the below will be helpful. First define a class to hold any data you need to send to the endpoint and use this as TInput on your ActionBlock, something like this:

   class EndpointArguments
   {
      // what information do you need to submit the request?
   }

use it like this:

var endpointActionBlock = new ActionBlock<EndpointArguments>(data =>
    {
       // This will be invoked for every item you post to the data block.
       // Invoke the endpoint using data and log the result
       // you can use a policy handler like Polly for catching and retrying failed requests, etc.
    },
    new ExecutionDataflowBlockOptions
    {
        MaxDegreeOfParallelism = 10, // how many concurrent requests do you want?
        EnsureOrdered = false // does the order matter?
        // check the other options offered by this class
    }
);


// Post values to the block.
while (moreValuesToProcess)
{
     endpointActionBlock.Post(new EndpointArguments { ... });
}

// Signal to the block we are done adding requests.  They will continue running in the background.
endpointActionBlock.Complete();

// Wait for completion in a try/catch block.
try
{
   await endpointActionBlock.Completion;
}
catch (AggregateException ae)
{
   // If an unhandled exception occurs during dataflow processing, all
   // exceptions are propagated through an AggregateException object.
   // You probably want to handle the exception in the callback / delegate to prevent this from happening
   ae.Handle(e =>
   {
      Console.WriteLine("Encountered {0}: {1}", e.GetType().Name, e.Message);
      return true;
   });
}