-1

I have a created a background job which processes users(ex: 2000 users, calling other apis and adding to database), I have divided users into few chunks (ex: 10 chunks) and processing them in tasks for each chunk.

I need to get progress of how many users getting processed from each individual task and so that I can sum and take it as progress.

How can I do that?

private void CallingMethod()
{
    List<Task> allTasks = new List<Task>();

    allTasks.Add(Task.Run(() => ProcessUsers(usersList1));
    allTasks.Add(Task.Run(() => ProcessUsers(usersList2));
    allTasks.Add(Task.Run(() => ProcessUsers(usersList3));

    await allTasks;

    int GetCountsFromTasks????
}

private async Task ProcessUsers(List<User> usersList)
{
    int processedUsers = 0;
    foreach(var user in usersList)
    {
        // Processing of users
    
        //End
        processedUsers++; // Need to send this count to calling method
                          // after each user completion
    }
    // send results object as well to calling method which contains usernames,
    // success/fail for each user at end of task
}
Theodor Zoulias
  • 34,835
  • 7
  • 69
  • 104
Sai Kiran
  • 55
  • 6
  • You probably want to raise events in the subtask, with status updates and subscribe to those in the parent thread... I've not got the time to write this up in more detail# – ScottishTapWater Feb 09 '23 at 21:17
  • 1
    Passing an `IProgress` delegate to the Tasks may do, or incrementing a shared counter (`Intelocked.Increment`), could be the *observable* property of a class. Much depends on what the caller is doing with this information – Jimi Feb 09 '23 at 21:30
  • Using it for updating in database and show on UI when someone requests it – Sai Kiran Feb 09 '23 at 21:40
  • Can you give little more code details or any link which i can refer to @Jimi – Sai Kiran Feb 09 '23 at 21:41
  • 1
    Not enough information. How is the calling method updating a database with *a number*? You're not trying to interact with the `List` you're passing to the Tasks, right? Why are you using `Task.Run()` instead of adding these Tasks - which appear to be async - to a collection and awaiting `Task.WhenAll()`? What kind of UI are you trying to update? Concurrent execution of Tasks that then need to all wait while another operation is in place (as updating a database) may not be perform as expected. Or are you planning on just updating the UI with the progress, then the database in the end? – Jimi Feb 09 '23 at 21:59
  • Why did you create separate lists of 10 elements out of 2000? Do you mean that you then need to run 200 concurrent Tasks? Not with `Task.Run()`, I assume? Why not use a single collection? I.e., what is the actual purpose of the split? – Jimi Feb 09 '23 at 22:01
  • I have divided into 10 tasks of 200 users each. Yes, as you said i was adding all these tasks into a list of tasks and doing task.whenall(allTasks); – Sai Kiran Feb 09 '23 at 22:03
  • But i want the progress to be sent to the calling method, which i will update into dynamodb for that request. If a user requests status, i will pull it from dynamo db and show it. As it is long running process, i want to show count of users processed for the user when he requests from UI – Sai Kiran Feb 09 '23 at 22:06
  • I can look at the options of pub sub as mentioned by @ScottishTapWater – Sai Kiran Feb 09 '23 at 22:19
  • Also is there anyway to create all these tasks in a for loop. I splitted them individually for now in the code as i was getting error, when i created tasks in a for loop – Sai Kiran Feb 09 '23 at 22:28
  • @SaiKiran - To be clear, I'm talking about C# events, not an external pubsub bus like RabbitMQ – ScottishTapWater Feb 09 '23 at 23:35
  • Could you correct the signature of the `CallingMethod`? Is it `async Task` or `async void`? – Theodor Zoulias Feb 10 '23 at 20:03

1 Answers1

0

Progress updates are normally done via IProgress<T>. There isn't much in the way of combining IProgress<T> instances, but you can build your own solution. Regarding results, those are normally returned from asynchronous methods, and can be retrieved from await allTasks if you do that.

Something like this:

private void CallingMethod()
{
  int usersProcessed1 = 0;
  int usersProcessed2 = 0;
  int usersProcessed3 = 0;
  var progress1 = new Progress<int>(value => usersProcessed1 = value);
  var progress2 = new Progress<int>(value => usersProcessed2 = value);
  var progress3 = new Progress<int>(value => usersProcessed3 = value);

  List<Task> allTasks = new List<Task>();
  allTasks.Add(Task.Run(() => ProcessUsers(usersList1, progress1));
  allTasks.Add(Task.Run(() => ProcessUsers(usersList2, progress2));
  allTasks.Add(Task.Run(() => ProcessUsers(usersList3, progress3));

  var results = await allTasks;
}

private async Task<IReadOnlyList<UserResult>> ProcessUsers(List<User> usersList, IProgress<int> progress)
{
  int processedUsers = 0;
  foreach(var user in usersList)
  {
    // Processing of users
    processedUsers++;
    progress?.Report(processedUsers);
  }

  return results;
}
Stephen Cleary
  • 437,863
  • 77
  • 675
  • 810