Each element of InputQueue in TransformBlock is overwritten whenever a record is read

Question

Purpose

I'm trying to create a pipeline where I read from a file one bytes record at a time and send it to a `BufferBlock', which append items in the buffer block. This is linked through the trivial LinkTo () method to TransformBlock <byte [], MyObject> which converts each record of bytes into the MyObject object. Below is the whole method that does all this:

    BufferBlock<byte[]> buffer = new BufferBlock<byte[]>();
    TransformBlock<byte[], MyObject> transform = new TransformBlock<byte[], MyObject>(bytes =>
    {
        return FromBytesTOMyObject(bytes);
    });

    private void ReadFileAndAppend()
    {
        buffer.LinkTo(transform, new DataflowLinkOptions() { PropagateCompletion = true });

        BinaryReader br = new BinaryReader(new FileStream("C:\\Users\\MyUser\\myFile.raw",FileMode.Open, FileAccess.Read, FileShare.Read));                                  
        int count;
        byte[] record = new byte[4000];

        // Post more messages to the block.
        Task.Run(async () =>
        {
            while ((count = br.Read(record, 0, record.Length)) != 0)
                await buffer.SendAsync(record);
            buffer.Complete();
        });
        transform.Completion.Wait();
        Console.WriteLine("");

Below the method that is called inside the TransformBlock:

static public MyObject FromBytesToMyObject(byte[] record)
    {
        MyObject object = new MyObject();
        object.counter = BitConverter.ToInt32(record, 0);
        object.nPoints = BitConverter.ToInt32(record, 4);

        for (int i = 0; i < object.nPoints; i++)
        {
            int index = i * 4;
            object.A[i] = BitConverter.ToSingle(record, index + 8);
        }
        return object;
    }

as can be seen from the FromBytesToMyObject () methodthere is a counter inside each record read. So each record never has a counter equal to another record (I also checked through a byte reader like HxD)

Problem

With this setting I thought that the reading and interpretation of the file took place smoothly. But going into debugging and inserting a breakpoint in the "while" after reading about 50 records or more, I noticed that in TransformBlock's OutputQueue groups of records with the same counter are queued, and therefore identical records.

Example:

Exact queue considering only the counters: 1,2,3,4,5,6,7,8,9,10.

Queue that I actually see in OutputQueue: 1,2,2,2,3,3,3,3,4,4,5,5 ....

Can you explain to me where I'm wrong?

The `while (br.BaseStream.Position != br.BaseStream.Length)` line looks sketchy. The completion of a stream-reading loop is usually determined by the `reader.Read` method returning a zero. Using a externally provided `byte[]` buffer looks even sketchier. God knows where else this buffer could be used concurrently... — Theodor Zoulias, Nov 11 '20 at 18:56
Without a [mcve] it's impossible to provide a good answer. That said, from the description it seems almost certain that you are using the same reference-type object instance with each block, and so of course when you modify that object, all other references to the same object show the same modification. — Peter Duniho, Nov 11 '20 at 19:30
As a side note, the `BufferBlock` is probably redundant. You could feed the `TransformBlock` directly, since it has its own internal input queue. — Theodor Zoulias, Nov 15 '20 at 03:17

score 0 · Accepted Answer · answered Nov 15 '20 at 03:14

0

You are reusing the same byte[] record over and over again, without any thread-safety considerations. No wonder that things are going south fast. If you want to ensure the correctness of the whole operation, you must use a different buffer each time:

while (true)
{
    var record = new byte[4000];
    var count = binaryReader.Read(record, 0, record.Length);
    if (count == 0) break;
    Array.Resize(ref record, count);
    await buffer.SendAsync(record);
}
buffer.Complete();

If you also care about the performance and don't want to overburden the garbage collector, you should look at the ArrayPool class. But be careful because this class offers new ways to shoot yourself in the foot.

answered Nov 15 '20 at 03:14

Theodor Zoulias

34,835
7
69
104

what is the `Array.Resize(ref record, count);` for, if the variable is reassigned every time? – NickMan Nov 16 '20 at 09:56
@NickMan the last `Read` will most probably read less than 4000 bytes, so you'll need some way to communicate the exact number of bytes read when passing the buffer to the `TransformBlock`. You'll have to either resize the buffer, or pass the count along with the buffer to the block. – Theodor Zoulias Nov 16 '20 at 10:57

Each element of InputQueue in TransformBlock is overwritten whenever a record is read

Purpose

Problem

1 Answers1