Concurrent i/o and processing for large data files in python

Question

I am going to write a python program that reads chunks from a file, processes those chunks and then appends the processed data to a new file. It will need to read in chunks as the files to process will generally be larger than the amount of ram available greatly simplified pseudocode, It will be something like this:

def read_chunk(file_object, chunksize):
    # read the data from the file object and return the chunk
    return chunk

def process_chunk(chunk):
    #process the chunk and return the processed data
    return data

def write_chunk(data, outputfile):
    # write the data tothe output file.

def main(file):
    # This will do the work
    for i in range(0, numberofchunks, chunksize):
        chunk = read_chunk(file_obj, chunksize)
        data = process_chunk(chunk)
        write_chunk(data, out_file)

What I'm wondering, is can I execute these 3 methods concurrently and how would that work?

I.e one thread to read the data, one thread to process the data and one thread to write the data. Of course, the reading thread would always need to be one 'step' ahead of the processing thread, which needs to be one step ahead of the writing thread...

What would be really great would be able to execute it concurrently and split it among processors...

More detail on the exact problem: I'll be reading data from a raster file using the GDAL library. This will read in chunks/lines into a numpy array. The processing will simply be some logical comparisons between the value of each raster cell and it's neighbours (which neighbour has a lower value than the test cell and which of those is the lowest). A new array of the same size (edges are assigned arbritary values) will be created to hold the result and this array will be written to a new raster file. I anticipate that the only other library than GDAL will be numpy, which could make this routine a good candidate for 'cythonising' aswell.

Any tips on how to proceed?

Edit:

I should point out that we have implemented similar things previously, and we know that the time spent processing will be significant compared to I/O. Another point is that the library we will use for reading the files (GDAL) will support concurrent reading...

Answer 1

coroutines for handling data pipelines? This template should get you started, in a way that minimizes memory profile. You could add a queuing and virtual 'fake-thread' manager to this, for multiple files.

#!/usr/bin/env python3

import time
from functools import wraps, partial, reduce

def coroutine(func_gen):
    @wraps(func_gen)
    def starter(*args, **kwargs):
        cr = func_gen(*args, **kwargs)
        _ = next(cr)
        return cr
    return starter


@coroutine
def read_chunk(file_object, chunksize, target):
    """
    read enless stream with a .read method
    """
    while True:
        buf = file_object.read(chunksize)
        if not buf:
            time.sleep(1.0)
            continue
        target.send(buf)

@coroutine
def process_chunk(target):
    def example_process(thing):
        k = range(100000000) # waste time and memory
        _ = [None for _ in k]
        value = str(type(thing))
        print("%s -> %s" % (thing, value))
        return thing

    while True:
        chunk = (yield)
        data  = example_process(chunk)
        target.send(data)

@coroutine
def write_chunk(file_object):
    while True:
        writable = (yield)
        file_object.write(writable)
        file_object.flush()


def main(src, dst):
    r = open(src, 'rb')
    w = open(dst, 'wb')

    g = reduce(lambda a, b: b(a),
               [w, write_chunk, process_chunk,
                partial(read_chunk, r, 16)]
              )
    while True:
        _ = next(g)

main("./stackoverflow.py", "retval.py")

Answer 2

My honest advice is not to worry about optimization right now (see premature optimization).

Unless you'll be doing a lot of operations (it doesn't seem so from your description), there's a very high chance that the I/O waiting time will be much, much larger than the processing time (i.e.: I/O is the bottleneck).

In that situation, doing the processing in multiple threads is useless. Splitting I/O and processing (as you suggested) will only buy you at most n*proc_time, with n being the number of times you process, and proc_time the processing time of each operation (not including I/O). If proc_time is much lower than the I/O time, you won't gain much.

I'd implement this serially at first, check I/O and CPU usage, and only then worry about optimization, if it seems it might be advantageous. You may also want to experiment with reading more chunks from the file at once (buffering).