0

I am inserting roughly 10k documents from a CSV file into a rethinkDB table; the code looks like this

print "inserting records",
with open('input.csv', 'rb') as csvfile:
    reader = csv.reader(csvfile, delimiter=',')
    for row in reader:
        document = # do some stuff to create the document
        insert_record(document)
        print ".",

As you can see after each insert a dot should be printed on screen to show the progress of the operation. Unfortunately what I see instead is:

  1. nothing happens for a few seconds
  2. the "inserting records" and a large number of dots is shown all at once
  3. nothing happens for a few seconds
  4. again a number of dots is shown all at once
  5. 3 - 4 repeat until all docuemnts are inserted

Why are the print commands "cached" and then done in batches and how can I fix it?

FLIR31207
  • 73
  • 1
  • 7
  • Not an answer to the question but something to consider: RethinkDB supports bulk inserts: [RethinkDB FAQ](http://www.rethinkdb.com/blog/answers-to-common-questions/#can-rethinkdb-do-bulk-inserts) – chucksmash Sep 06 '15 at 23:42

1 Answers1

0

Usually when printing to stdout, the output will be buffered and then flushed when it grows large enough. Try adding sys.stdout.flush() after the print statement, and the output should be smoother.

Tryneus
  • 371
  • 2
  • 4