How to write file chunk to disk as soon as the range of bytes are completed by a thread

Question

continuing from this: https://codereview.stackexchange.com/questions/

I discovered that holding bytes of data in a list may be expensive and may slow down system performance, so I think it is better to write the data to disk as soon as the range of that thread is completed and so rest of the chunk finishing should write the chunk of range coming from next thread finishing in queue..

def main(url=None, splitBy=6):
    start_time = time.time()
    if not url:
        print "Please Enter some url to begin download."
        return

    fileName = url.split('/')[-1]
    sizeInBytes = requests.head(url, headers={'Accept-Encoding': 'identity'}).headers.get('content-length', None)
    print "%s bytes to download." % sizeInBytes
    if not sizeInBytes:
        print "Size cannot be determined."
        return
    threads = []
    lock = threading.Lock()

    byteRanges = buildRange(int(sizeInBytes), splitBy)
    for idx in range(splitBy):
        bufTh = SplitBufferThread(url, byteRanges[idx])
        bufTh.daemon = True
        bufTh.start()
        threads.append(bufTh)

    print "--- %s seconds ---" % str(time.time() - start_time)

    with open(fileName, 'w+') as fh:
        for t in threads:
            t.join()
            fh.write(t.data)
        fh.flush()

    print "Finished Writing file %s" % fileName

if __name__ == '__main__':
    main(url)

So How to write file chunk to disk as soon as the range of bytes are completed by a thread ?

@JamieCockburn : How to write file chunk to disk as soon as the range of bytes are completed by a thread? — Ciasto piekarz, Jul 06 '14 at 12:08
Each thread would need to be given an open file handle and an offset into the file to write the downloaded data. You would probably also need to provide a Lock to prevent the file from being written-to simultaneously by multiple threads. — martineau, Jul 06 '14 at 12:41
how do I lock the waiting threads as per the current logic ? — Ciasto piekarz, Jul 06 '14 at 13:23

How to write file chunk to disk as soon as the range of bytes are completed by a thread

0 Answers0