Multi-threaded loop through collection with C#

Question

I have seen lots of examples on how to do this (read through lots of threads on this site), but I'm having a very hard time getting something that works without locking up the UI thread AND allows me to control the number of threads executing at a time.

I have this method, which loops through the collection and calls an async method to process the items, but it locks up the UI...

    async Task ProcessItems()
    {
        using (var semaphore = new SemaphoreSlim(5))
        {
            var tasks = MyCollection.Select(async (MyItem) =>
            {
                await semaphore.WaitAsync();
                try
                {
                    await ProcessItem(MyItem);
                }
                finally
                {
                    semaphore.Release();
                }
            }).ToArray();

            await Task.WhenAll(tasks);
        }
    }

The 'ProcessItem' method calls another method, which makes some API calls over the internet to upload some files.

This has been working, but I have no control over how many threads are executed at a time...

    foreach (MyItemClass MyItem in MyCollection)
    {
        MyItemClass tmp = logItem;
        Thread thread = new Thread(() => ProcessItem(tmp));
        thread.Start();
    }

EDIT: Here's the ProcessItem method...

    public async Task<string> ProcessItem(MyItemClass MyItem)
    {
        MyItem.Status = "Transferring";

        HelperClass.UploadFileOrFolder(
            siteUrl,
            MyItem.DocumentLibrary,
            MyItem.RootFolder,
            srcRootPath,
            MyItem.SourcePath);

        MyItem.Status = "Transferred";

        return "Transferred";
    }

And here is the UploadFileorFolder method, which is using the SharePoint.Client.File.SaveBinaryDirect method to upload a file to SharePoint...

    public static void UploadFileOrFolder(string siteUrl, string libraryName, string rootFolder, string path, string file)
    {
        ClientContext ctx = GetSPContext(siteUrl);

        using (fs = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize: 4096, useAsync: true))
        {
            Microsoft.SharePoint.Client.File.SaveBinaryDirect(ctx, targetFileUrl, fs, true);
        }
    }

If ProcessItems method locks up UI, then you're doing something wrong in ProcessItem method. I suspect it does synchronous work, post the code of `ProcessItem`. — Sriram Sakthivel, Feb 26 '15 at 14:54
You can save your created threads int a collection/array/list and then check which is still alive. — LPs, Feb 26 '15 at 15:00
Why do you need fined grain control over the number of threads? In virtually all situations, it's best to allow the thread pool or another framework method handle allocating threads; it's almost certainly going to do it more efficiently than you. — Servy, Feb 26 '15 at 15:03
The code you've provided is entirely asynchronous. If it's blocking the UI thread, it's due to code not shown, either above or below that method on the call stack. — Servy, Feb 26 '15 at 15:04
1) create # of threads you want processing at a time 2) slice your collection by this number 3) give each thread a slice to process 4) grab a beer. What's the issue? — , Feb 26 '15 at 15:10
@Fubak: Read the warning the compiler gives you on the `ProcessItem` method. — Stephen Cleary, Feb 26 '15 at 15:23
Creating an unbounded number of threads can severely impact your whole system and most certainly look like *your* UI is blocked. — Peter Ritchie, Feb 26 '15 at 15:26
@PeterRitchie He's not creating an unbounded number of threads. In fact, he never has more than one thread running at a time. — Servy, Feb 26 '15 at 15:29
@StephenCleary: It says, 'This async method lacks await operators and will run synchronously.' Can you help me understand where I'm missing await operators? This is where I start to get confused. — Fubak, Feb 26 '15 at 15:32
@servy I read the `foreach` that created a `Thread` object and called `Start`. — Peter Ritchie, Feb 26 '15 at 15:36
@Fubak: See [Servy's answer](http://stackoverflow.com/a/28746257/263693). **If** your API (`SaveBinaryDirect`) has an asynchronous equivalent (i.e., `SaveBinaryDirectTaskAsync` or `SaveBinaryDirectAsync`), then you can use `await` to call the asynchronous version. Otherwise, you'll have to block thread pool threads (i.e., `Task.Run`). — Stephen Cleary, Feb 26 '15 at 15:39
@StephenCleary: Doesn't look like it has an Async version. This MSDN article was the closest I could get to finding a solution. https://msdn.microsoft.com/en-us/library/office/ee857094(v=office.14).aspx#SP2010ClientOM_Asynchronous_Processing What would the Task.Run solution look like? — Fubak, Feb 26 '15 at 15:51

score 4 · Accepted Answer · answered Feb 26 '15 at 15:23

4

Your ProcessItem method is marked as async, and it returns a Task, which indicates it's supposed to be asynchronous, but it is not in fact asynchronous. it will execute entirely synchronously. async merely allows the use of await in a method, and does nothing to actually make a method asynchronous. You are currently getting a compiler warning informing you of this fact.

If the API you're using exposes an actually asynchronous method to upload the file you're trying to upload, you should be using that instead of the synchronous version. If it doesn't, then while it's undesirable, you'll need to use Task.Run (or an equivalent) to run the synchronous method in a thread pool thread asynchronously.

answered Feb 26 '15 at 15:23

Servy

202,030
26
332
449

Can you add some examples of how my code would look using Task.Run? – Fubak Feb 26 '15 at 15:52
@Fubak You wrap the method that you want to be run in a thread pool in a `Task.Run` call and `await` it. – Servy Feb 26 '15 at 15:58
@Fubak or remove `async` and just return the Task. [however, do read this](http://blogs.msdn.com/b/pfxteam/archive/2012/03/24/10287244.aspx) as a warning – default Feb 26 '15 at 17:13

Multi-threaded loop through collection with C#

1 Answers1