We have a n number of task whic is running sequentially now. We want to convert this to a parallel concurrent task. There are multiple ways to acieve this. Lets see few of those here.

Sequential task example

var urls = new [] { 
        "https://github.com/nishanthatn/nishanthatn",
        "https://nishanth.dev/",
        "https://nishanth.dev/index.php/about/",
        "https://nishanth.dev/index.php/2023/03/",
        "https://nishanth.dev/index.php/2019/07/",
        "https://nishanth.dev/index.php/category/angular/",
        "https://nishanth.dev/index.php/2023/03/04/vs-code-angular-debug-configuration/",
        "https://nishanth.dev/index.php/blog/" 
};
var client = new HttpClient();
foreach(var url in urls)
{
    var html = await client.GetStringAsync(url);
    Console.WriteLine($"retrieved {html.Length} characters from {url}");
}

To parallelize this, we could just turn every single download into a separate Task with Task.Run and wait for them all to complete, but what if we wanted to limit the number of concurrent downloads? Let’s say we only want 3 downloads to happen at a time.

In this trivial example, the exact number might not matter too much, but it’s not hard to imagine a situation in which you would want to avoid too many concurrent calls to a downstream service.

Solution 1 : Concurrent Queue

var maxThreads = 3;
var q = new ConcurrentQueue<string>(urls);
var tasks = new List<Task>();
for(int n = 0; n < maxThreads; n++)
{
    tasks.Add(Task.Run(async () => {
        while(q.TryDequeue(out string url)) 
        {
            var html = await client.GetStringAsync(url);
            Console.WriteLine($"retrieved {html.Length} characters from {url}");
        }
    }));
}
await Task.WhenAll(tasks);

I like this approach as its conceptually simple. But it can be a bit of a pain if we are still generating more work to do while we’ve started processing work as the reader threads could exit too early.

Solution 2 : Parallel For Each

var options = new ParallelOptions() { MaxDegreeOfParallelism = maxThreads };
Parallel.ForEach(urls, options, url =>
    {
        var html = client.GetStringAsync(url).Result;
        Console.WriteLine($"retrieved {html.Length} characters from {url}");
    });

This solution should only be used if you have a synchronous method you want to perform in parallel. 

Solution 3 : Parallel For Each Async

var options = new ParallelOptions() { MaxDegreeOfParallelism = maxThreads };
await Parallel.ForEachAsync(urls, options, async (url, token) =>
{
    var html = await client.GetStringAsync(url);
    Console.WriteLine($"retrieved {html.Length} characters from {url}");

});

Parallel.ForEachAsync is available from .NET 6.