Dequeue a collection to write to disk until no more items left in collection-CodePudding

So, I have a list of file shares. I then need to obtain ALL folders in these file shares. This is all the "easy" stuff I have done. Ultimately after some logic, I am adding an object into a collection.

All the while this is happening in the background using Async/Await and Tasks, I want to be able to have another thread/task spin up so it can keep going through the collection and write data to disk.

Now, for each folder, I obtain security information about that folder. There will be at LEAST 1 item. But for each item, I add this information into a collection (for each folder).

I want to write to disk in a background until there are no more folders to iterate through and the job is complete.

I was thinking of using a BlockingCollection however this code does smell and ultimately does not close the file because of the while(true) statement.

private static BlockingCollection<DirectorySecurityInformation> AllSecurityItemsToWrite = new BlockingCollection<DirectorySecurityInformation>();

if (sharesResults.Count > 0)
{
    WriteCSVHeader();

    // setup a background task which will dequeue items to write.
    var csvBGTask = Task.Run(async () =>
    {
        using (var sw = new StreamWriter(FileName, true))
        {
            sw.AutoFlush = true;
            while (true)
            {
                var dsi = AllSecurityItemsToWrite.Take();
                await sw.WriteLineAsync("... blah blah blah...");
                await sw.FlushAsync();
            }
        }
    });
    allTasks.Add(csvBGTask);
}

foreach(var currentShare in AllShares)
{
   var dirs = Directory.EnumerateDirectories(currentShare .FullName, "*", SearchOption.AllDirectories);
   foreach(var currentDir in dirs) { // Spin up a task in the BG and run to do some security analysis and add to the AllSecurityItemsToWrite collection }
}

This is at its simplest but core example. Any ideas? I just want to keep adding on the background task and have another task just dequeue and write to disk until there are no more shares to go through (shareResults).

CodePudding user response：

Recommand to use Channel.

Channel<DirectorySecurityInformation> ch = 
    Channel.CreateUnbounded<DirectorySecurityInformation>();

Write

var w = ch.Writer;
foreach(var dsi in DSIs)
    w.TryWrite(dsi);
w.TryComplete();

Read

public async void ReadTask()
{
    var r = ch.Reader;
    using (var sw = new StreamWriter(filename, true))
    {
        await foreach(var dsi in r.ReadAllAsync())
            sw.WriteLine(dsi);
    }
}

CodePudding user response：

while (true)
{
    var dsi = AllSecurityItemsToWrite.Take();
    //...
}

Instead of using the Take method, it's generally more convenient to consume a BlockingCollection<T> with the GetConsumingEnumerable method:

foreach (var dsi in AllSecurityItemsToWrite.GetConsumingEnumerable())
{
    //...
}

This way the loop will stop automatically when the CompleteAdding method is called, and the collection is empty.

But I agree with shingo that the BlockingCollection<T> is not the correct tool in this case, because your workers are running on an asynchronous context. A Channel<T> should be preferable, because it can be consumed without blocking a thread.