Memory issue in TPL Dataflow implementation of IO read write operation

Question

I have tried to implement Read write operation using File IO operations and encapsulated those operations into TransformBlock so as to make these operation thread safe instead of using locking mechanism.

But the problem is that when I try to write even 5 files parallely there is a memory out of exception and on using this implementation it is blocking UI thread. The implementation is done in Windows Phone project. Please suggest what is wrong in this implemantation.

File IO Operation

public static readonly IsolatedStorageFile _isolatedStore = IsolatedStorageFile.GetUserStoreForApplication();
public static readonly FileIO _file = new FileIO();
public static readonly ConcurrentExclusiveSchedulerPair taskSchedulerPair = new ConcurrentExclusiveSchedulerPair();
public static readonly ExecutionDataflowBlockOptions exclusiveExecutionDataFlow 
    = new ExecutionDataflowBlockOptions
{
    TaskScheduler = taskSchedulerPair.ExclusiveScheduler,
    BoundedCapacity = 1
};

public static readonly ExecutionDataflowBlockOptions concurrentExecutionDataFlow 
    = new ExecutionDataflowBlockOptions
{
    TaskScheduler = taskSchedulerPair.ConcurrentScheduler,
    BoundedCapacity = 1
};

public static async Task<T> LoadAsync<T>(string fileName)
{
    T result = default(T);

    var transBlock = new TransformBlock<string, T>
       (async fName =>
       {
           return await LoadData<T>(fName);
       }, concurrentExecutionDataFlow);

    transBlock.Post(fileName);

    result = await transBlock.ReceiveAsync();

    return result;
}

public static async Task SaveAsync<T>(T obj, string fileName)
{
    var transBlock = new TransformBlock<Tuple<T, string>, Task>
       (async tupleData =>
       {
          await SaveData(tupleData.Item1, tupleData.Item2);
       }, exclusiveExecutionDataFlow);

    transBlock.Post(new Tuple<T, string>(obj, fileName));

    await transBlock.ReceiveAsync();
}

MainPage.xaml.cs Usage

private static string data = "vjdsskjfhkjsdhvnvndjfhjvkhdfjkgd"
private static string fileName = string.Empty;
private List<string> DataLstSample = new List<string>();
private ObservableCollection<string> TestResults = new ObservableCollection<string>();
private static string data1 = "hjhkjhkhkjhjkhkhkjhkjhkhjkhjkh";
List<Task> allTsk = new List<Task>();
private Random rand = new Random();
private string  fileNameRand
{
    get
    {
        return rand.Next(100).ToString();
    }
}

public MainPage()
{
    InitializeComponent();

    for (int i = 0; i < 5; i ++)
    {
        DataLstSample.Add((i % 2) == 0 ? data : data1);
    }

}

private void Button_Click(object sender, RoutedEventArgs e)
{
    AppIsolatedStore_TestInMultiThread_LstResultShouldBeEqual();
}

public async void AppIsolatedStore_TestInMultiThread_LstResultShouldBeEqual()
{
    TstRst.Text = "InProgress..";
    allTsk.Clear();

    foreach(var data in DataLstSample)
    {
        var fName = fileNameRand;

        var t = Task.Run(async () =>
        {
            await AppIsolatedStore.SaveAsync<string>(data, fName);
        });

        TestResults.Add(string.Format("Writing file name: {0}, data: {1}", fName, data));
        allTsk.Add(t);
    }

    await Task.WhenAll(allTsk);

    TstRst.Text = "Completed..";
}

Save and Load data Async

        /// <summary>
        /// Load object from file
        /// </summary>
        private static async Task<T> LoadData<T>(string fileName)
        {

            T result = default(T);

            try
            {
                if (!string.IsNullOrWhiteSpace(fileName))
                {
                    using (var file = new IsolatedStorageFileStream(fileName, FileMode.OpenOrCreate, _isolatedStore))
                    {
                        var data = await _file.ReadTextAsync(file);

                        if (!string.IsNullOrWhiteSpace(data))
                        {
                            result = JsonConvert.DeserializeObject<T>(data);
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                //todo: log the megatron exception in a file
                Debug.WriteLine("AppIsolatedStore: LoadAsync : An error occured while loading data : {0}", ex.Message);
            }
            finally
            {

            }

            return result;
        }


        /// <summary>
        /// Save object from file
        /// </summary>
        private static async Task SaveData<T>(T obj, string fileName)
        {
            try
            {
                if (obj != null && !string.IsNullOrWhiteSpace(fileName))
                {
                    //Serialize object with JSON or XML serializer
                    string storageString = JsonConvert.SerializeObject(obj);

                    if (!string.IsNullOrWhiteSpace(storageString))
                    {
                        //Write content to file
                        await _file.WriteTextAsync(new IsolatedStorageFileStream(fileName, FileMode.Create, _isolatedStore), storageString);
                    }
                }
            }
            catch (Exception ex)
            {
                //todo: log the megatron exception in a file
                Debug.WriteLine("AppIsolatedStore: SaveAsync : An error occured while saving the data : {0}", ex.Message);
            }
            finally
            {
            }
        }

Edit:

The reason that it is having memory exception is because of one reason that data string which i took is too huge. The string is link: http://1drv.ms/1QWSAsc

But the second problem is that if i add small data also then it is blocking UI thread. Is code doing any task on UI tread?

VMAtm · Accepted Answer · 2015-12-24T16:48:26.460

No, you use the concurrent pair which uses the default thread pool for it's tasks, and you instantiate tasks with Run method, so the problem isn't here. But the code you have here has two main threats:

var transBlock = new TransformBlock<string, T>
   (async fName =>
   {
       // process file here
   }, concurrentExecutionDataFlow);

You really shouldn't create transBlock each time. The main idea of TPL Dataflow is that you are creating the blocks once and use them after that. So you should refactor your app to low the number of blocks you are instantiating, otherwise this is not the case TPL Dataflow should been used.

Another threat in your code is that you are explicitly block the thread!

// Right here
await Task.WhenAll(allTsk);
TstRst.Text = "Completed..";

Calling an await for task from a async void method from a synchronious event handler blocks the thread, as by default it captures the synchronization context. First of all, async void should be avoided. Second, if you are async, you should be async all the way, so event handler should be async too. Third, you may use a continuation for your task for update your UI or use current synchronization context.

So, your code should be something like this:

// store the sync context in the field of your form
SynchronizationContext syncContext = SynchronizationContext.Current;

// avoid the async void :)
public async Task AppIsolatedStore_TestInMultiThread_LstResultShouldBeEqual()

// make event handler async - this is the only exception for the async void use rule from above
private async void Button_Click(object sender, RoutedEventArgs e)

// asynchronically wait the result without capturing the context
await Task.WhenAll(allTsk).ContinueWith(
  t => {
    // you can move out this logic to main method
    syncContext.Post(new SendOrPostCallback(o =>
        {
            TstRst.Text = "Completed..";
        }));
  }
);

I am exploring ways to make IO operations function thread-safe without using locking mechanism. As to avoid side-effects of locking. As you say that I should reduce the number of block creation or use a diff approach. Can you suggest a way to make this better or a new way which I can explore more. — Balraj Singh, Dec 23 '15 at 11:02
@BalrajSingh TPL Dataflow still uses blocks internally. Manual `lock` statements are much more readable, and much more efficient — VMAtm, Dec 23 '15 at 11:47
In the final example, `ConfigureAwait` is wrong - this wouldn't compile. Also, `ContinueWith` shouldn't be used. If the op is using TPL Dataflow, a more idiomatic solution would be a final `ActionBlock` with a UI `TaskScheduler`. — Stephen Cleary, Dec 24 '15 at 16:28
@StephenCleary Thank you for addition! I forgot about bracers. — VMAtm, Dec 24 '15 at 16:50

score 0 · Answer 2 · answered Dec 23 '15 at 05:53

Have you tried playing around with the BoundedCapacity parameter on the ExecutionDataflowBlockOptions ? The Introduction to TPL mentions this about block capacity:

[...] bounding is useful in a dataflow network to avoid unbounded memory growth. This can be very important for reliability reasons if there’s a possibility that producers could end up generating data much faster than the consumers could process it...

Id suggest trying to use this option this limit the queuing of processed items and see if it helps with your memory issues

I have set BoundingCapacity to 1 still the issue persist. – Balraj Singh Dec 23 '15 at 08:19 — Balraj Singh, Dec 23 '15 at 08:19

Memory issue in TPL Dataflow implementation of IO read write operation

2 Answers2