Reactive Extensions: Creating a pipeline with Rx that works with files

Question

I have a process pipeline with three steps:

video to images: I have a video that is converted to still images (frames)
frames to zip file: When all the frames in a video have been processed, I should create a Zip File with them.
zip file => Upload to FTP

It involves two disposables: the video capture and the zip file.

How could I handle it using Rx? I don't know how to start.

Start by breaking the code down into exactly those three steps. Next, ensure that each of those methods returns an `IObservable` to represent the async nature of transcodeing/zipping/uploading. Finally, use SelectMany/Concat/Merge/etc to create an Rx pipeline using those 3 methods. — Lee Campbell, Jun 21 '16 at 00:23
Thanks, Lee! The thing is that I don't know if it really makes sense to create an IObservabe, since this abstraction doesn't really represent data, but an abstraction to create a .zip. If I manage to create this kind of observable, I would receive a stream of disposed ZipArchives. — SuperJMN, Jun 21 '16 at 12:06
More generally, I feel that a projections that end up hitting the filesytem end up being tricky. Maybe, a way to do this is to project a token (string) that represents the name of the video and with Do, to create a Zip and passing the token to the next step in the pipeline. Finally, the subscriber will receive the token to know which file to upload. — SuperJMN, Jun 21 '16 at 12:17

score 6 · Accepted Answer · answered Jun 21 '16 at 14:35

Do you have to pass along the raw objects from each step. It should be fine that the video capture or the zip file are "disposed", because I am sure there would be some side effect form the action (a MemoryStream, a file written to disk etc.). You can just pass on the pointers (Uri's?) to the results of each action to the next part of the pipeline?

void Main()
{
    var video = new Uri("https://www.youtube.com/watch?v=Tp5mRlHwZ7M");
    var query = from frames in TranscodeVideoToImages(video)
        from zipFile in ZipFiles(frames)
        from uploadLocation in UploadFile(zipFile)
        select uploadLocation;
    query.Subscribe(...)
}
private IObservable<Uri[]> TranscodeVideoToImages(Uri imageSource)
{
    //Do some long running (async) work here.
    // Save work to disk
    // Return location of saved work
}
private IObservable<Uri> ZipFiles(Uri[] files)
{
    //Run the zip process on all of the files
    //  Return the location of the zip file
}
private IObservable<Uri> UploadFile(Uri source)
{
    //Upload the File. 
    //Probably as simple as as task based operation with .ToObservable()
}

Reactive Extensions: Creating a pipeline with Rx that works with files

1 Answers1