How to parallelize asynchronous integration work that includes JSON deserialization?-CodePudding

One question about parallel programming.

I need to POST 2 APIs (one after another) to get a Order list. First one is for getting token..

After I got the list, I need to POST 3 API's (one after another) to integrate these Orders.

These 3 API's don't accept arrays, so I need to send one by one. I cannot send batch data. With 1 thread it only integrates 10 orders in a minute. I need more performance. How can I run foreach part in parallel?

using System.Net.Http;
using System.Text;
using System.Text.Json;

namespace Order_Integrator
{
    public class Program
    {

        static readonly HttpClient client = new HttpClient();

        static async Task Main()
        {

            //Auth
            var connectResponse = await client.PostAsync(connectUrl, connectContent);
            var connectResponseString = await connectResponse.Content.ReadAsStringAsync();
            var connect = JsonSerializer.Deserialize<connectResponse>(connectResponseString);
            
            var token = connect.Token;
            //Get Order List
            var orderResponse = await client.PostAsync(orderUrl, orderContent);
            var orderResponseString = await orderResponse.Content.ReadAsStringAsync();
            var orders = JsonSerializer.Deserialize<orderResponse>(orderResponseString);

            foreach (var order in orders)
            {
            
                //Get Order Details
                //Generate getOrderDetailsContent with order 
                var getOrderDetailsResponse = await client.PostAsync(getOrderDetailsUrl, getOrderDetailsContent);
                var getOrderDetailsResponseString = await getOrderDetailsResponse.Content.ReadAsStringAsync();
                var getOrderDetails = JsonSerializer.Deserialize<getOrderDetailsResponse>(getOrderDetailsResponseString);

                //Create Order
                //Generate createOrderContent with GetOrderDetails
                var createOrderResponse = await client.PostAsync(createOrderUrl, createOrderContent);
                var createOrderResponseString = await createOrderResponse.Content.ReadAsStringAsync();
                var createOrder = JsonSerializer.Deserialize<createOrderResponse>(createOrderResponseString);

                //Create Log
                //Generate createLogContent with CreateOrderResponse
                var createLogResponse = await client.PostAsync(createLogUrl, createLogContent);
                var createLogResponseString = await createLogResponse.Content.ReadAsStringAsync();
                var createLog = JsonSerializer.Deserialize<createLogResponse>(createLogResponseString);
            }
        }
    }
}

I saw examples for voids, but it didn't work for Task.

CodePudding user response：

You can use Parallel.ForEachAsync

 var options = new ParallelOptions()
{
    MaxDegreeOfParallelism = 20
};
await Parallel.ForEachAsync(orders, options, async (OrderNumber, ct) => {

var getOrderDetailsResponse = await client.PostAsync(getOrderDetailsUrl, getOrderDetailsContent);
    
});

CodePudding user response：

Since you're using asynchronous code, you should be using asynchronous concurrency, not (multithreaded) parallelism. Parallel.ForEachAsync will do both, but it's not available on older runtimes.

You can do just asynchronous concurrency by using Select and then Task.WhenAll:

var tasks = orders.Select(async order =>
{
  //Get Order Details
  //Generate getOrderDetailsContent with order 
  var getOrderDetailsResponse = await client.PostAsync(getOrderDetailsUrl, getOrderDetailsContent);
  var getOrderDetailsResponseString = await getOrderDetailsResponse.Content.ReadAsStringAsync();
  var getOrderDetails = JsonSerializer.Deserialize<getOrderDetailsResponse>(getOrderDetailsResponseString);

  //Create Order
  //Generate createOrderContent with GetOrderDetails
  var createOrderResponse = await client.PostAsync(createOrderUrl, createOrderContent);
  var createOrderResponseString = await createOrderResponse.Content.ReadAsStringAsync();
  var createOrder = JsonSerializer.Deserialize<createOrderResponse>(createOrderResponseString);

  //Create Log
  //Generate createLogContent with CreateOrderResponse
  var createLogResponse = await client.PostAsync(createLogUrl, createLogContent);
  var createLogResponseString = await createLogResponse.Content.ReadAsStringAsync();
  var createLog = JsonSerializer.Deserialize<createLogResponse>(createLogResponseString);

  return createLog;
}).ToList();
await Task.WhenAll(tasks);