Efficient way to run method by multiple thread(task)

Question

I am looking for the best way to run a method with multiple threads.

Here is a method I'd like to run

public async Task<IActionResult> GenerateById(int id)
{
    //make a web request and save response to database using EF.
    return Json(new { success = true, message = "Worked" });
}

I need to run this method for Id 1 to 40 and currently, it's done by hangfire with 10 jobs.

In startup.cs

recurringJobManager.AddOrUpdate("GenerateAll1",
                () => serviceProvider.GetService<IGenerator>().GenerateAll(1), "0 * * * *");
recurringJobManager.AddOrUpdate("GenerateAll2",
                () => serviceProvider.GetService<IGenerator>().GenerateAll(2), "0 * * * *");
recurringJobManager.AddOrUpdate("GenerateAll3",
                () => serviceProvider.GetService<IGenerator>().GenerateAll(3), "0 * * * *");
....
recurringJobManager.AddOrUpdate("GenerateAll10",
                () => serviceProvider.GetService<IGenerator>().GenerateAll(10), "0 * * * *");

This is GenerateAll

public async Task GenerateAll(int mod)
{
    if (mod == 10)
    {
        mod = 0;
    }
    List<DataSetting> listsToUpdate = _unitOfWork.DataSetting.GetAll(r => r.Id%10 == mod ).ToList();
    foreach (var list in listsToUpdate )
    {   
       await GenerateById(list.Id); 
    }
}

So each jobs handle 4 ids(eg..GenerateAll1 runs for Id 1,11,21 and 31) But I found that this is inefficient. Because when GenerateById(11) takes long, GenerateById(21) and GenerateById(31) won't be running until GenerateById(11) is finished even though GenerateAll2 job is already finished.

What I want to do is that run just 1 job by hangfire like

recurringJobManager.AddOrUpdate(
       "GenerateAll1",
                () => serviceProvider.GetService<IGenerator>().GenerateAll(), "0 * * * *");

and GeneratAll() method creates 10 threads and each of threads grab Id and run GenerateById method. And when it's done, it runs GenerateById with an id that hasn't been generated. So all 10 threads are working until all data settings are generated.

Can I please get any suggestions on this situation?

Thanks

why don't you create one `Task` for each id, start them all at once, and then just `await` the whole collection? no need to mess around with threads here, i think. — Franz Gleichmann, Nov 09 '20 at 06:10
Would something like `Parallel.For(0, 40, x => GenerateById(x));` work? — Jerry, Nov 09 '20 at 06:13
OP needs to give more information. Is this on an Asp.net application? Do you need to call ` GenerateById(int id)` 40 times per request? etc. — Hasan Emrah Süngü, Nov 09 '20 at 06:16

Jerry · Answer 1 · 2020-11-10T06:05:44.843

1

Parallel.For is useful for what you are trying to do. It's unclear what your intentions for the result of GenerateById() are, so I will leave that to you to figure out.

From the docs:

Executes a for loop in which iterations may run in parallel.

recurringJobManager.AddOrUpdate(
    "GenerateAll1",
        () => serviceProvider.GetService<IGenerator>().GenerateAll(), "0 * * * *");

public async Task GenerateAll()
{
    Parallel.For(0, _unitOfWork.DataSetting.Length,
        new ParallelOptions() { MaxDegreeOfParallelism = 10 },
        x => GenerateById(_unitOfWork.DataSetting[x]).Result());
}

public async Task<IActionResult> GenerateById(int id)
{
    ...
    return Json(new { success = true, message = "Worked" });
}

edited Nov 10 '20 at 06:05

answered Nov 09 '20 at 06:31

Jerry

1,477
7
14

Hi Jerry, Thanks for your suggestion. I should have mentioned that mehtod GenerateById(). It make web requests and save response data to Database,so it's done by EF. And I got an error that "A second operation started on this context before a previous operation completed" . Do you have any ideas that I can use parallel with method use EF? – dbenbyeon Nov 10 '20 at 05:16
If you want all of the threads to access the same instance of a DbContext then you would have to use a lock so only one thread was accessing it at a time. If the operations on the context are fast compared to your web requests then you'll still have large benefits of doing this parallelly. – Jerry Nov 10 '20 at 05:31

Efficient way to run method by multiple thread(task)

1 Answers1