What is the fastest and most efficient ways to make a large number of C# WebRequests?-CodePudding

I have a list of URLs (thousands), I want to asynchronously get page data from each URL as fast as possible without putting extreme load on the CPU.

I have tried using threading but it still feels quite slow:

public static ConcurrentQueue<string> List = new ConcurrentQueue<string>(); //URL List (assume I added them already)

public static void Threading()
{
    for(int i=0;i<100;i  ) //100 threads
    {
        Thread thread = new Thread(new ThreadStart(Task));
        thread.Start();
    }
}

public static void Task()
{
    while(!(List.isEmpty))
    {
        List.TryDequeue(out string URL);
        //GET REQUEST HERE
    }
}

Is there any better way to do this? I want to do this asynchronously but I can't figure out how to do it, and I don't want to sacrifice speed or CPU efficiency to do so.

Thanks :)

CodePudding user response：

You are best off using HttpClient which allows async Task requests.

Just store each task in a list, and await the whole list. To prevent too many requests at once, wait for any single one to complete if there are too many, and remove the completed one from the list.

const int maxDegreeOfParallelism = 100;

static HttpClient _client = new HttpClient();

public static async Task GetAllUrls(List<string> urls)
{
    var tasks = List<Task>(urls.Count);

    using var sem = new SemaphoreSlim()
    foreach (var url in urls)
    {
        if (tasks.Count == maxDegreeOfParallelism)    // this prevents too many requests at once
            tasks.Remove(await Task.WhenAny(tasks));

        tasks.Add(GetUrl(url));
    }
    await Task.WhenAll(tasks);
}

private static async Task GetUrl(string url)
{
    using var response = await _client.GetAsync(url);
    // handle response here
    var responseStr = await response.Content.ReadAsStringAsync(); // whatever
    // do stuff etc
}

CodePudding user response：

You should use Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:

public static IObservable<(string url, string content)> GetAllUrls(List<string> urls) =>
    Observable
        .Using(
            () => new HttpClient(),
            hc =>
                from url in urls.ToObservable()
                from response in Observable.FromAsync(() => hc.GetAsync(url))
                from content in Observable.FromAsync(() => response.Content.ReadAsStringAsync())
                select (url, content));

That allows you to consume the results in a couple of ways.

You can process them as they get produced:

IDisposable subscription =
    GetAllUrls(urlsx).Subscribe(x => Console.WriteLine(x.content));

Or you can get all of them produced and then await the full results:

(string url, string content)[] results = await GetAllUrls(urlsx).ToArray();