I would like to read a stream and buffer it's output so that a consumer could read it before the producer has finish full reading of the stream. example, read from an Http stream and forward to another ... This using the .net4.5 TPL library. I found out the Nito AsyncCollection and below is what I wrote. but I am wondering if this is all correct because when I debug, even using a long string to test it and even when debuging from asp.net vnext pipeline, it does readfrom and write to in sequence... How could I be sure it has the correct behaviour in NON debug mode ?
using Nito.AsyncEx;
using System;
using System.Collections.Concurrent;
using System.IO;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
namespace Absyla.Core
{
public class PipeStream
{
private AsyncCollection<byte[]> _chunks = new AsyncCollection<byte[]>(1);
public async Task ReadFrom(Stream s)
{
try
{
byte[] buffer;
int bytesRead = 0;
int totalBytesRead = 0;
do
{
long buffsize = Math.Min(s.Length - totalBytesRead, 1L);
buffer = new byte[buffsize];
bytesRead = await s.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead > 0)
{
int readId = System.Environment.CurrentManagedThreadId;
await _chunks.AddAsync(buffer);
totalBytesRead += bytesRead;
}
}
while (bytesRead > 0);
}
finally
{
_chunks.CompleteAdding();
}
}
public async Task WriteTo(Stream s)
{
while (await _chunks.OutputAvailableAsync())
{
int writeId = System.Environment.CurrentManagedThreadId;
byte[] buffer = await _chunks.TakeAsync();
await s.WriteAsync(buffer, 0, buffer.Length);
}
}
}
}
here is a test
[Fact]
public async Task ShouldBufferLittleString()
{
String testString = "test in";
PipeStream ps = new PipeStream();
MemoryStream ms2 = new MemoryStream();
await ps.ReadFrom(testString.AsStream());
await ps.WriteTo(ms2);
string result = ms2.ReadFromStart();
result.ShouldBe(testString);
}
note : the AsStream() method is an extension that returns a stream object from a String.
As I found it simple, I wonder if this is really correct and if it really does the job I want...
UPDATE : changing the following:
private AsyncCollection<byte[]> _chunks = new AsyncCollection<byte[]>();
to
private AsyncCollection<byte[]> _chunks = new AsyncCollection<byte[]>(2);
made the throttling work. but my test blocks. and it does work when I do this:
public async Task ShouldBufferLittleString()
{
String testString = "test in";
PipeStream ps = new PipeStream();
MemoryStream ms2 = new MemoryStream();
var readTask = ps.ReadFrom(testString.AsStream());
var writeTask = ps.WriteTo(ms2);
await Task.WhenAll(readTask, writeTask);
string result = ms2.ReadFromStart();
result.ShouldBe(testString);
}
but in my code I cannot always have reference to the read and write task as it can be called from different location in the code.
I tried to change to BufferBlock, but had the same issue.
UPDATE 2 : I changed the pipestream code a little to add throtling and an buffer of 1 just for test.
in the test, when I await immediately it blocks, but when I create the task and I await them after, the test passes.
// this blocks:
await ps.ReadFrom(testString.AsStream());
await ps.WriteTo(ms2);
//this passes
var read = ps.ReadFrom(testString.AsStream());
var write = ps.WriteTo(ms2);
await read;
await write;
