Parallel.ForEach returning before object's method which makes rate limited API calls

Question

I am working on a plugin for a program that needs to make API calls, I was previously making them all synchronously which, well it worked, was slow.

To combat this I am trying to make the calls asynchronous, I can make 10 per second so I was trying the following:

Parallel.ForEach(
    items.Values,
    new ParallelOptions { MaxDegreeOfParallelism = 10 },
    async item => {
        await item.UpdateMarketData(client, HQOnly.Checked, retainers);
        await Task.Delay(1000);
    }
);

client is an HttpClient object and the rest is used to build the API call or for the stuff done to the result of the API call. Each time item.UpdateMarketData() is called 1 and only 1 API call is made.

This code seems to be finishing very quickly and as I understand it, the program should wait for a Parallel.ForEach() to complete before continuing.

The data that should be set by item.UpdateMarketData() is not being set either. In order to make sure, I have even set MaxDegreeOfParallelism = 1 and the Delay to 3 seconds and it still finished very quickly despite having ~44 items to go though. Any help would be appreciated.

UpdateMarketData() is included below just in case it is relevant:

public async Task UpdateMarketData(TextBox DebugTextBox,HttpClient client,
    bool HQOnly, List<string> retainers)
{
    HttpResponseMessage sellers_result = null;
    try
    {
        sellers_result = await client.GetAsync(String.Format(
            "www.apiImCalling/items/{0}?key=secretapikey", ID));
    }
    catch (Exception e)
    {
        System.Windows.Forms.MessageBox.Show(
            String.Format("{0} Exception caught.", e));
        sellers_result = null;
    }
    var results = JsonConvert.DeserializeObject<RootObjectMB>(
        sellers_result.Content.ReadAsStringAsync().Result);
    int count = 0;
    OnMB = false;
    LowestOnMB = false;
    LowestPrice = int.MaxValue;
    try
    {
        foreach (var x in results.Prices)
        {
            if (x.IsHQ | !(HQOnly && RequireHQ))
            {
                count++;
                if (count == 1)
                {
                    LowestPrice = x.PricePerUnit;
                }
                if (retainers.Contains(x.RetainerName))
                {
                    Retainer = x.RetainerName;
                    OnMB = true;
                    Price = x.PricePerUnit;
                    if (count == 1)
                    {
                        LowestOnMB = true;
                    }
                }
                if (LowestPrice == x.PricePerUnit
                    && x.RetainerName != Retainer)
                {
                    LowestOnMB = false;
                }
            }
        }
    }
    catch (Exception e)
    {
        System.Windows.Forms.MessageBox.Show(
            String.Format("{0} Exception caught.", e));
    }
}

I havn't had a chance to try it yet, its after 230 am here, and I am not really able of focusing on it, I will absolutely try to get this working tomorrow — sleepless, Feb 14 '19 at 07:33

score 3 · Accepted Answer · answered Feb 14 '19 at 13:37

async doesn't work with Parallel. One is asynchronous, the other is parallel, and these are two completely different styles of concurrency.

To restrict the concurrency of asynchronous operations, use SemaphoreSlim. E.g.:

var mutex = new SemaphoreSlim(10);
var tasks = items.Values.Select(item => DoUpdateMarketData(item)).ToList();
await Task.WhenAll(tasks);

async Task DoUpdateMarketData(Item item)
{
  await mutex.WaitAsync();
  try
  {
    await item.UpdateMarketData(client, HQOnly.Checked, retainers);
    await Task.Delay(1000);
  }
  finally { mutex.Release(); }
}

You may find my book helpful; this is covered in recipe 11.5.

score 0 · Answer 2 · answered Feb 14 '19 at 06:37

0

Rather then parallel.for loop , you can make use of Task and wait for all task to complete.

var tasks = new List<Task>();
foreach (var val in items.Values)
    tasks.Add(Task.Factory.StartNew(val.UpdateMarketData(client, HQOnly.Checked, retainers)));

 try
 {
    // Wait for all the tasks to finish.
    Task.WaitAll(tasks.ToArray());
    //make use of WhenAll method if you dont want to block thread, and want to use async/await 

    Console.WriteLine("update completed");
  }
  catch (AggregateException e)
  {
     Console.WriteLine("\nThe following exceptions have been thrown by WaitAll(): (THIS WAS EXPECTED)");
    for (int j = 0; j < e.InnerExceptions.Count; j++)
    {
        Console.WriteLine("\n-------------------------------------------------\n{0}", e.InnerExceptions[j].ToString());
    }
  }

answered Feb 14 '19 at 06:37

Pranay Rana

175,020
35
237
263

How does this limit the number of calls per second? It seems the OP has a limitation of 10 calls per second. – Peter Bons Feb 14 '19 at 07:40
@PeterBons - he has to add logic related to that – Pranay Rana Feb 14 '19 at 07:47
I think you're trying to iterate through a collection that is not always containing a correct value (correct in a business context). Your foreach (var x in results.Prices) will not wait until you get a sellers_result from GetAsync(). Try to log the content of your sellers_result just to make sure you always get it right. – Soufiane Tahiri Feb 14 '19 at 08:35

Parallel.ForEach returning before object's method which makes rate limited API calls

2 Answers2